-- Logs begin at Wed 2022-09-07 03:21:19 UTC, end at Wed 2022-09-07 04:02:03 UTC. -- Sep 07 03:46:06.735196 dv-cinder-856002-2-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 07 03:46:06.743097 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 07 03:46:06.745112 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: *** Starting uWSGI 2.0.18-debian (64bit) on [Wed Sep 7 03:46:06 2022] *** Sep 07 03:46:06.745112 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 07 03:46:06.745112 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 07 03:46:06.745112 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: nodename: dv-cinder-856002-2-iscsi Sep 07 03:46:06.745112 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: machine: x86_64 Sep 07 03:46:06.745112 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: clock source: unix Sep 07 03:46:06.745112 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: pcre jit disabled Sep 07 03:46:06.745566 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: detected number of CPU cores: 4 Sep 07 03:46:06.745566 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: current working directory: / Sep 07 03:46:06.745566 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: detected binary path: /usr/bin/uwsgi-core Sep 07 03:46:06.745566 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: your processes number limit is 47680 Sep 07 03:46:06.745566 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: your memory page size is 4096 bytes Sep 07 03:46:06.745566 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: detected max file descriptor number: 2048 Sep 07 03:46:06.745566 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: lock engine: pthread robust mutexes Sep 07 03:46:06.745566 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: thunder lock: enabled Sep 07 03:46:06.745566 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 07 03:46:06.745566 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 07 03:46:06.786176 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: Python main interpreter initialized at 0x55cfa055f9c0 Sep 07 03:46:06.786176 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: python threads support enabled Sep 07 03:46:06.786176 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: your server socket listen backlog is limited to 100 connections Sep 07 03:46:06.786176 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: your mercy for graceful operations on workers is 90 seconds Sep 07 03:46:06.786574 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: mapped 403077 bytes (393 KB) for 2 cores Sep 07 03:46:06.786654 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: *** Operational MODE: preforking *** Sep 07 03:46:06.786897 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: *** uWSGI is running in multiple interpreter mode *** Sep 07 03:46:06.786897 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: spawned uWSGI master process (pid: 81378) Sep 07 03:46:06.787099 dv-cinder-856002-2-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 07 03:46:06.788193 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: spawned uWSGI worker 1 (pid: 81391, cores: 1) Sep 07 03:46:06.788193 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: spawned uWSGI worker 2 (pid: 81392, cores: 1) Sep 07 03:46:06.788193 dv-cinder-856002-2-iscsi devstack@keystone.service[81378]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 07 03:46:11.055754 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 07 03:46:11.055754 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: from cryptography.utils import int_from_bytes Sep 07 03:46:11.056555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 07 03:46:11.056555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: from cryptography.utils import int_from_bytes Sep 07 03:46:11.057003 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 07 03:46:11.057003 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: from cryptography.utils import int_from_bytes Sep 07 03:46:11.057138 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 07 03:46:11.057138 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: from cryptography.utils import int_from_bytes Sep 07 03:46:11.106055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 07 03:46:11.106266 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 07 03:46:11.106406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 07 03:46:11.106501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] command line args: [] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 07 03:46:11.106610 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 07 03:46:11.106719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 07 03:46:11.106807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] command line args: [] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 07 03:46:11.106885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ================================================================================ {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 07 03:46:11.107070 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 07 03:46:11.107155 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] admin_token = **** {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.107252 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ================================================================================ {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 07 03:46:11.107381 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] config_dir = [] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.107510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] admin_token = **** {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.107589 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.107741 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] config_source = [] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.107823 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] config_dir = [] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.107924 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] debug = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.108007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.108111 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] config_source = [] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.108194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.108370 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] debug = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.108491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.108581 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.108673 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.108740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.108888 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.109002 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.109135 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.109252 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.109371 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] list_limit = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.109476 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.109579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] log_config_append = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.109673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] list_limit = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.109768 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.109875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] log_config_append = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.109979 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] log_dir = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.110086 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.110224 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] log_file = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.110329 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] log_dir = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.110329 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] log_file = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.110542 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.110648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.110815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.110961 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.111077 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.111189 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.111312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.111312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.111533 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.111533 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.111758 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.111885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.112003 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.112117 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.112228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.112349 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.112474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.112598 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.112720 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.112825 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.112945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.113066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.113188 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.113321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.113436 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.113560 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.113675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.113784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.113898 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.114026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.114190 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.27/identity {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.114310 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.27/identity {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.114456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] publish_errors = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.114573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] publish_errors = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.114688 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.114807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.114922 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.115049 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.115165 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.115321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.115446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.115582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.115707 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.115823 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.115944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] standard_threads = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.116061 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] standard_threads = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.116181 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.116305 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.116427 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.116542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.116657 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.116777 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.116899 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] use_journal = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.117066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] use_journal = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.117192 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] use_json = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.117307 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] use_json = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.117421 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] use_stderr = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.117536 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] use_stderr = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.117650 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] use_syslog = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] use_syslog = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.117899 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.118014 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 07 03:46:11.118146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.118263 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.118379 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.118511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.118636 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.118752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.118872 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.118987 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.119102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.119225 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.119341 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.119455 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.119576 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.119692 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.119813 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.119940 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.120057 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.120172 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.120287 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.120402 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.120515 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.120641 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.120763 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.120880 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.120994 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.121108 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.121229 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.121358 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.121474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.121589 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.121703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.121818 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.121932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.122054 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.122198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.122313 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.122429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.122542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.122740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.122865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.122982 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.123096 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.123210 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.123325 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.123450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.123571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.123686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.123800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.123915 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.124036 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.124164 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.124280 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.124398 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.124515 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.124629 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.124750 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.124874 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.124996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.125111 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.125225 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.125339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.125454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.125577 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.125693 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.125813 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.125929 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.126043 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.126183 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.126825 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.126950 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.127066 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.127180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.127301 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.127460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.127588 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.127704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.127819 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.127933 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.128047 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.128161 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.128161 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.128387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.128502 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.128617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.128738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.128853 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.129049 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.129168 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.129283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.129397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.129511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.129626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.129750 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.129872 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.129994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.130131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.130249 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.130374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.130500 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.130617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.130732 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.130847 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.130961 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.131075 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.131182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.131286 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.131383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.131479 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.131573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.131669 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.131773 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.131871 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.131982 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.132113 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.132241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.132356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.132472 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.132587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.132703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.132819 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.132933 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.133061 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.133179 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.133296 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.133412 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.133536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.133673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.133803 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.133922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.134050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.134218 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.134337 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.134455 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.134571 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.134696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.134825 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.134944 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.135066 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.135189 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.135295 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.135410 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.135534 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.135534 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.135749 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.135866 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.135982 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.136103 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.136228 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.136334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.136437 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.136551 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.136666 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.136781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.136907 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.137025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.137140 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.137256 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.137371 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.137486 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.137609 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.137726 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.137841 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.137957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.138072 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.138202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.138331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.138456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.138572 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.138688 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.138803 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.138918 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.139044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.139169 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.139292 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.139409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.139524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.139646 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.139771 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.139888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.140004 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.140120 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.140236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.140351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.140542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.140662 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.140777 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.140893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.141009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.141134 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.141261 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.141390 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.141508 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.141624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.141740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.141855 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.142608 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.142747 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.142867 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.143136 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.143267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.143387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.143504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.143630 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.143748 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.143865 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.143980 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.144103 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.144227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.144354 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.144479 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.144479 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.144722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.144840 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.144984 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.145111 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] auth.external = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.145230 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] auth.external = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.145355 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.145475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.145592 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.145722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.145842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.145966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.146092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] auth.password = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.146237 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] auth.password = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.146357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] auth.token = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.146476 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] auth.token = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.146593 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.146710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.146835 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.146954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.147061 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.147177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.147294 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.147411 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.147537 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.147663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.147782 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.147900 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.148016 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.148133 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.148262 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] credential.caching = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.148381 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] credential.caching = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.148498 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.148616 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.148733 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.148850 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.148976 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.149094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.149212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.149330 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.149447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.149564 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.149690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.149808 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.149926 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.150043 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.150185 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.150304 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.150439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.150559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.150679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.150796 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.150912 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.151029 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.151155 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.151274 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.151391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.151508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.151634 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.151754 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.151880 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.151999 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.152116 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] federation.caching = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.152233 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] federation.caching = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.152349 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.152466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.152595 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.152714 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.152833 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.152950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.153067 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.153196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.153322 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.153441 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.153441 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.153662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.153780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.153906 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.154032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.154164 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.154294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.154422 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.154545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.154672 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.154791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.154909 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.155027 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity.caching = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.155176 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity.caching = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.155294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.155485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.155607 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.155725 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.155843 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.155961 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.156078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.156204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.156323 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.156446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.156565 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.156682 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.156799 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.156924 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.157043 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.157160 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.157277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.157393 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.157703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.158026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.158195 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.158317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.158435 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.158552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.158669 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.158800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.158920 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.159039 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.159156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.159281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.159400 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.159526 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.159646 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.159763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.159880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.159996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.160114 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.160239 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.160358 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.160474 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.160591 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.160708 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.160825 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.160825 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.161061 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.161180 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.161297 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.161413 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.161530 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.161655 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.161773 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.161890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.162007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.162139 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.162258 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.162387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.162507 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.162624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.162740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.162863 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.162990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.163109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.163227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.163344 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.163474 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.163639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.163794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.163914 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.164031 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.164147 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.164270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.164388 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.164513 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.164631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.164748 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.164869 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.164984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.165099 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.165223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.165341 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.165456 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.165571 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.165686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.165801 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.165925 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.166042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.166204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.166323 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.166440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.166555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.166684 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.166802 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.166918 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.167033 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.167147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.167263 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.167389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.167514 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.167667 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.167787 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.167904 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.168020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.168145 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.168264 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.168381 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.168499 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.168615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.168732 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.168857 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.168975 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.169092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.169208 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.169325 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.169442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.169442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.169672 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.169790 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.169906 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.170023 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.170153 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.170285 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.170405 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.170523 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.170640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.170758 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.170874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.171070 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.171203 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.171322 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.171439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.171555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.171673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.171798 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.171917 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.172034 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.172150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.172267 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.172383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.172508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.172626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.172750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.172868 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.172985 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.173101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.173279 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.173401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.173518 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.173635 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.173753 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.173869 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.173994 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.174140 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.174261 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.174378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.174506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.174625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.174758 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.174878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.174995 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.175112 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.175229 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.175344 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.175470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.175589 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.175706 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.175824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.175940 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.176056 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.176181 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.176299 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.176417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.176533 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.176649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] resource.caching = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.176765 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.176897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.176897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.177119 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.177236 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.177352 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] resource.caching = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.177468 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.177593 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.177712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.177828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.177946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.178062 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.178266 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.178397 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.178517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.178634 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.178751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.178867 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.179056 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] role.caching = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.179177 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.179295 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] role.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.179411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.179528 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.179645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.179770 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.179889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] role.caching = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.180008 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.180125 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] role.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.180241 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.180358 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.180483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.180602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.180719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.180836 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.180953 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.181071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.181195 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.181314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.181432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.181549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.181666 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.181782 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.181907 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.182025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.182161 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.182279 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.182397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.182513 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.182643 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.182763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.182763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.182983 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.183099 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.183218 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.183342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.183461 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.183461 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.183679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.183795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.183911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.184036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.184154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.184272 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.184388 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.184504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.184629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.184751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.184868 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.184976 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.185091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.185207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.185333 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.185451 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.185568 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.185685 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.185801 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.185917 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.186048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.186180 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.186301 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.186418 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.186534 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.186651 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.186780 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.186900 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.187017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.187133 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.187249 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.187367 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] token.caching = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.187492 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.187610 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.187727 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.187843 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.187966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.188084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.188209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] token.caching = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.188328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.188446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.188562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.188679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.189288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.189431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.189559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.189702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.189873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.189996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.190159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.190295 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.190416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.190534 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.190650 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.190766 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.190882 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.191006 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.191125 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.191242 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.191358 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.191474 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.191592 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.191721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.191853 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.191971 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.192089 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.192205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.192321 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.connection = **** {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.192437 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.192567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.192695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.192822 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.192941 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.193058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.193174 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.connection = **** {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.193290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.193406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.193534 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.193654 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.193771 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.193887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.194003 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.194003 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.194256 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.194256 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.194479 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.194598 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.194717 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.194839 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.194965 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.195083 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.195200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.195317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.195433 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.195549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.195675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.195794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.195911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.196027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.196144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.196144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.196370 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.196489 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.196605 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.196722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81391) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 07 03:46:11.196840 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.196840 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.196840 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.196840 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 07 03:46:11.196840 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81392) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 07 03:46:11.223021 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.223250 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.223476 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.223604 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.223749 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.223974 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.231778 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.232311 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.261390 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.261616 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.261760 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.261944 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.319284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.322447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.368581 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.368816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.369670 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.369909 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.524025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.524208 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.524283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.524495 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.524607 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.524675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.524737 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.524891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.524974 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.525055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.525119 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.525285 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.548409 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.548530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.548692 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.548859 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.548943 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.549081 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.549163 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.549312 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.549407 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.549472 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.549534 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.549693 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.549776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.549917 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.560677 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.560853 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.573139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.573375 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.573465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.573618 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.573703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.573835 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.573917 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.574048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.574171 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.574315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.574398 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.574531 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81392) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.574611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.574688 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81391) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 07 03:46:11.591869 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.592083 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 07 03:46:11.592254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.592429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.592517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 07 03:46:11.592670 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 07 03:46:11.592813 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.592895 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.593026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 07 03:46:11.593178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.593272 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.593406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.593489 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.593666 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.593754 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.593999 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.603855 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.604136 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.604279 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.604414 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.604569 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.604728 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.604863 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.604998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.611684 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.611896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.614611 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.614821 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.614966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.615100 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.615235 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.615384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.615531 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.615706 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.615846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.615980 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.616113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.616257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.616392 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.616526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.616659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.616793 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.616945 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.617080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.617224 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.617359 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.617492 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.617625 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.639981 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.640139 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.641452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.641600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.641735 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.641892 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.646998 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.647145 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.647279 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.647412 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.647556 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.647693 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.653889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.654040 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.654204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.654341 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.660340 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.660512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.660648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.660781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.660914 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.661059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.661194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.661374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.661509 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.661641 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.661793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.661929 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.662062 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.662228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.662365 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.662498 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.677733 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.677961 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.678108 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.678279 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.681821 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.682021 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.685030 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.685215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.685359 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.685520 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.689770 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.689927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.692855 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.693008 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.696060 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.696232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.696371 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.696505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.696638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.696772 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.696914 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.697051 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.697185 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.697319 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.697469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.697622 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.713835 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.713994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.714153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.714295 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.714428 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.714573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.714708 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.714842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.722051 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.722234 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.723612 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.723809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.725145 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.725289 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.725423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.725557 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.728124 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.728276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.728412 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.728545 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.728678 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.728841 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.728978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.729113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.729322 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.729468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.740646 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.740806 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.740944 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.741078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.741211 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.741369 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.741505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.741646 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.741835 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.741988 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.742167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.742311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.742446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.742578 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.758473 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.758986 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.761382 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.761879 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.764311 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.764771 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.765827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.766248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.767964 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.768245 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.768401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.768609 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.768766 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.769012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.775025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.775376 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.776488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.776815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.779425 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.779705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.779847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.780103 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.780323 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.780469 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.780611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.780742 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.788458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.788622 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.788770 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.788905 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.789036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.789174 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.795898 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81391) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.796062 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81392) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 07 03:46:11.796197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.796340 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.796483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.796617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.796747 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.796886 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.797019 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.797148 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.797278 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.797409 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.797548 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.797690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.797841 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.797973 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.798647 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.798811 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.799017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.799231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.799454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.799454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.799730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.799862 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.800022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.800153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81391) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.800283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.800283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81392) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 07 03:46:11.834896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55cfa055f9c0 pid: 81391 (default app) Sep 07 03:46:11.835497 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55cfa055f9c0 pid: 81392 (default app) Sep 07 03:46:11.838147 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf85d1ca-ed91-4c36-9de2-0122da4188c4 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:11.838437 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf85d1ca-ed91-4c36-9de2-0122da4188c4 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:11.838708 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf85d1ca-ed91-4c36-9de2-0122da4188c4 None None] PATH_INFO: `/v3` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:11.842864 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 1/1] 10.222.0.27 () {50 vars in 791 bytes} [Wed Sep 7 03:46:11 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:18.644455 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f253285c-f2e6-4bee-9927-cf34d13d5aa3 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:18.644977 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f253285c-f2e6-4bee-9927-cf34d13d5aa3 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:18.644977 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f253285c-f2e6-4bee-9927-cf34d13d5aa3 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:18.649298 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 1/2] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:18 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:46:18.654529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef64dfc1-6372-4ef2-9b29-655d8dead7fb None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:18.654967 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef64dfc1-6372-4ef2-9b29-655d8dead7fb None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:18.655103 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef64dfc1-6372-4ef2-9b29-655d8dead7fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:18.727229 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: INFO dbcounter [None req-ef64dfc1-6372-4ef2-9b29-655d8dead7fb None None] Registered counter for database keystone Sep 07 03:46:18.746569 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG oslo_db.sqlalchemy.engines [None req-ef64dfc1-6372-4ef2-9b29-655d8dead7fb None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81391) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 07 03:46:18.752891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writer thread running {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 07 03:46:18.940800 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG passlib.handlers.bcrypt [None req-ef64dfc1-6372-4ef2-9b29-655d8dead7fb None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81391) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 07 03:46:18.941102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG passlib.handlers.bcrypt [None req-ef64dfc1-6372-4ef2-9b29-655d8dead7fb None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81391) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 07 03:46:18.999164 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-ef64dfc1-6372-4ef2-9b29-655d8dead7fb None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:19.000752 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-ef64dfc1-6372-4ef2-9b29-655d8dead7fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:19.130780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 2/3] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:18 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 477 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 07 03:46:19.586038 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f253285c-f2e6-4bee-9927-cf34d13d5aa3 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:19.655792 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: INFO dbcounter [None req-f253285c-f2e6-4bee-9927-cf34d13d5aa3 None None] Registered counter for database keystone Sep 07 03:46:19.677937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG oslo_db.sqlalchemy.engines [None req-f253285c-f2e6-4bee-9927-cf34d13d5aa3 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81392) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 07 03:46:19.684130 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writer thread running {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 07 03:46:19.803469 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f253285c-f2e6-4bee-9927-cf34d13d5aa3 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:19.807826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d824183-f5c8-44b5-8363-604c65b58ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '388f90b795764a26a3cfac5c43c50a76', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:19.808456 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d824183-f5c8-44b5-8363-604c65b58ad7 admin admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:19.808742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d824183-f5c8-44b5-8363-604c65b58ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:19.809018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d824183-f5c8-44b5-8363-604c65b58ad7 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:19.817437 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d824183-f5c8-44b5-8363-604c65b58ad7 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:19.869682 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: WARNING keystone.common.rbac_enforcer.enforcer [None req-4d824183-f5c8-44b5-8363-604c65b58ad7 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 07 03:46:19.869992 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d824183-f5c8-44b5-8363-604c65b58ad7 admin admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:19.879513 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 2/4] 10.222.0.27 () {58 vars in 1324 bytes} [Wed Sep 7 03:46:19 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 07 03:46:19.885460 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef64dfc1-6372-4ef2-9b29-655d8dead7fb None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:19.900822 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef64dfc1-6372-4ef2-9b29-655d8dead7fb None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:19.905202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431b98b8-b74a-4b55-947b-e83b08f85854 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '388f90b795764a26a3cfac5c43c50a76', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:19.906342 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431b98b8-b74a-4b55-947b-e83b08f85854 admin admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:19.906624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431b98b8-b74a-4b55-947b-e83b08f85854 admin admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:19.906894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431b98b8-b74a-4b55-947b-e83b08f85854 admin admin] PATH_INFO: `/v3/projects/388f90b795764a26a3cfac5c43c50a76` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:19.909504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431b98b8-b74a-4b55-947b-e83b08f85854 admin admin] RBAC: Authorizing `identity:get_project(project_id=388f90b795764a26a3cfac5c43c50a76)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:19.961827 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: WARNING keystone.common.rbac_enforcer.enforcer [None req-431b98b8-b74a-4b55-947b-e83b08f85854 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 07 03:46:19.962188 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431b98b8-b74a-4b55-947b-e83b08f85854 admin admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:19.966786 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 3/5] 10.222.0.27 () {56 vars in 1207 bytes} [Wed Sep 7 03:46:19 2022] GET /identity/v3/projects/388f90b795764a26a3cfac5c43c50a76 => generated 344 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:46:22.107125 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b849094-849f-4713-ad59-2ace6c75c037 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:22.107745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b849094-849f-4713-ad59-2ace6c75c037 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:22.107745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b849094-849f-4713-ad59-2ace6c75c037 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:22.111899 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 3/6] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:22 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:22.117514 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a2ba5c-d832-4d7d-b875-bd9503348cde None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:22.117901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a2ba5c-d832-4d7d-b875-bd9503348cde None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:22.118069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a2ba5c-d832-4d7d-b875-bd9503348cde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:22.138002 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-a4a2ba5c-d832-4d7d-b875-bd9503348cde None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:22.139560 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-a4a2ba5c-d832-4d7d-b875-bd9503348cde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:22.147964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 4/7] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:22 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 07 03:46:22.414158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b849094-849f-4713-ad59-2ace6c75c037 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:22.426546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b849094-849f-4713-ad59-2ace6c75c037 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:22.430807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48455c19-1087-4be3-b992-1872ed042a0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '388f90b795764a26a3cfac5c43c50a76', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:22.431408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48455c19-1087-4be3-b992-1872ed042a0a admin admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:22.431690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48455c19-1087-4be3-b992-1872ed042a0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:22.431968 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48455c19-1087-4be3-b992-1872ed042a0a admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:22.440291 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48455c19-1087-4be3-b992-1872ed042a0a admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:22.442049 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48455c19-1087-4be3-b992-1872ed042a0a admin admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:22.450758 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 4/8] 10.222.0.27 () {58 vars in 1324 bytes} [Wed Sep 7 03:46:22 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 07 03:46:22.456988 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a2ba5c-d832-4d7d-b875-bd9503348cde None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:22.465999 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a2ba5c-d832-4d7d-b875-bd9503348cde None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:22.470795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1194429e-22ef-4ad0-ae30-b575a173b5b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '388f90b795764a26a3cfac5c43c50a76', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:22.471421 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1194429e-22ef-4ad0-ae30-b575a173b5b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:22.471702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1194429e-22ef-4ad0-ae30-b575a173b5b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:22.471976 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1194429e-22ef-4ad0-ae30-b575a173b5b9 admin admin] PATH_INFO: `/v3/users/c81176b6299a493294b869b497bad7f4` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:22.473747 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1194429e-22ef-4ad0-ae30-b575a173b5b9 admin admin] RBAC: Authorizing `identity:get_user(user_id=c81176b6299a493294b869b497bad7f4)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:22.475615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1194429e-22ef-4ad0-ae30-b575a173b5b9 admin admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:22.477136 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 5/9] 10.222.0.27 () {56 vars in 1198 bytes} [Wed Sep 7 03:46:22 2022] GET /identity/v3/users/c81176b6299a493294b869b497bad7f4 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:24.705799 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51f391c-35ef-4b53-9c94-5e9d63139efd None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:24.706362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51f391c-35ef-4b53-9c94-5e9d63139efd None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:24.706362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51f391c-35ef-4b53-9c94-5e9d63139efd None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:24.707721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 5/10] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:46:24.712923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832c95be-bbe1-48ca-ad24-bb98f186fbda None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:24.713338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832c95be-bbe1-48ca-ad24-bb98f186fbda None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:24.713471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832c95be-bbe1-48ca-ad24-bb98f186fbda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:24.731978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-832c95be-bbe1-48ca-ad24-bb98f186fbda None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:24.733142 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-832c95be-bbe1-48ca-ad24-bb98f186fbda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:24.742828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8e65e7-8ec6-4f78-803c-f31b83374e6e None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:24.743110 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8e65e7-8ec6-4f78-803c-f31b83374e6e None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:24.743384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8e65e7-8ec6-4f78-803c-f31b83374e6e None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:24.744725 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 6/11] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:46:24.749751 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec79f88f-811b-4c50-8c62-3da889d8fd09 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:24.750033 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec79f88f-811b-4c50-8c62-3da889d8fd09 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:24.750334 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec79f88f-811b-4c50-8c62-3da889d8fd09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:24.792484 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG passlib.handlers.bcrypt [None req-ec79f88f-811b-4c50-8c62-3da889d8fd09 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81392) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 07 03:46:24.792863 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG passlib.handlers.bcrypt [None req-ec79f88f-811b-4c50-8c62-3da889d8fd09 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81392) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 07 03:46:24.814511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 6/12] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:24.842818 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-ec79f88f-811b-4c50-8c62-3da889d8fd09 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:24.844408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-ec79f88f-811b-4c50-8c62-3da889d8fd09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:24.946572 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 7/13] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:46:25.084736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832c95be-bbe1-48ca-ad24-bb98f186fbda None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:25.109332 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:25.192460 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832c95be-bbe1-48ca-ad24-bb98f186fbda None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:25.216953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec79f88f-811b-4c50-8c62-3da889d8fd09 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:25.226480 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e85a4b-9f16-46ee-834d-6d2d2c9c9de5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:25.227124 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e85a4b-9f16-46ee-834d-6d2d2c9c9de5 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:25.227402 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e85a4b-9f16-46ee-834d-6d2d2c9c9de5 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:25.227677 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e85a4b-9f16-46ee-834d-6d2d2c9c9de5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:25.239581 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e85a4b-9f16-46ee-834d-6d2d2c9c9de5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:25.248766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-64e85a4b-9f16-46ee-834d-6d2d2c9c9de5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:46:25.248766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:25.248766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:25.248766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:25.248766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:25.248766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:25.248766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:25.248766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:25.248766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:25.248766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:25.248766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:25.248766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:25.248766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:25.248766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:25.248766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:25.248766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:25.249816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:25.249816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:25.249816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:25.249816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:25.249816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:25.249816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:25.249816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:25.249816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:25.249816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:25.249816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:25.249816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:25.249816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:25.249816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:25.249816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:25.249816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:25.249816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:25.255024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:25.255024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:25.255024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:25.255024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:25.255024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:25.255024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:25.255024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:25.255024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:25.255024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:25.255024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:25.255024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:46:25.255024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:46:25.255024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 7/14] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:46:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:25.258574 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e85a4b-9f16-46ee-834d-6d2d2c9c9de5 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:25.320285 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec79f88f-811b-4c50-8c62-3da889d8fd09 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:25.322605 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:25.339571 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:25.358483 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eae79e1-e8ee-491d-bae3-133559a861be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:25.359355 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eae79e1-e8ee-491d-bae3-133559a861be None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:25.359639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eae79e1-e8ee-491d-bae3-133559a861be None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:25.359913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eae79e1-e8ee-491d-bae3-133559a861be None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:25.360020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e85a4b-9f16-46ee-834d-6d2d2c9c9de5 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:25.394880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80288e65-4bed-43ae-b777-b742b55aec83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:25.395490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80288e65-4bed-43ae-b777-b742b55aec83 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:25.395769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80288e65-4bed-43ae-b777-b742b55aec83 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:25.396040 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80288e65-4bed-43ae-b777-b742b55aec83 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:25.396705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80288e65-4bed-43ae-b777-b742b55aec83 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:25.397007 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80288e65-4bed-43ae-b777-b742b55aec83 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:25.398429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80288e65-4bed-43ae-b777-b742b55aec83 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:25.410343 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 8/15] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:46:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:46:25.415962 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80288e65-4bed-43ae-b777-b742b55aec83 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:25.429953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eae79e1-e8ee-491d-bae3-133559a861be None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:25.431520 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eae79e1-e8ee-491d-bae3-133559a861be None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:25.514205 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80288e65-4bed-43ae-b777-b742b55aec83 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:25.530562 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 8/16] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:46:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 07 03:46:25.536280 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eae79e1-e8ee-491d-bae3-133559a861be None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:25.538953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:25.550319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541b996d-544e-4236-ab84-1b62e1591c93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:25.550957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541b996d-544e-4236-ab84-1b62e1591c93 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:25.551235 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541b996d-544e-4236-ab84-1b62e1591c93 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:25.551503 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541b996d-544e-4236-ab84-1b62e1591c93 None admin] PATH_INFO: `/v3/users/c81176b6299a493294b869b497bad7f4` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:25.552630 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541b996d-544e-4236-ab84-1b62e1591c93 None admin] RBAC: Authorizing `identity:get_user(user_id=c81176b6299a493294b869b497bad7f4)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:25.554142 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541b996d-544e-4236-ab84-1b62e1591c93 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:25.555727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 9/17] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:25 2022] GET /identity/v3/users/c81176b6299a493294b869b497bad7f4 => generated 247 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:46:25.561012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541b996d-544e-4236-ab84-1b62e1591c93 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:25.577292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:25.638908 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eae79e1-e8ee-491d-bae3-133559a861be None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:25.662526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541b996d-544e-4236-ab84-1b62e1591c93 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:25.673454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60da06c0-f058-4870-aceb-771126eae974 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:25.674414 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60da06c0-f058-4870-aceb-771126eae974 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:25.674698 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60da06c0-f058-4870-aceb-771126eae974 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:25.674974 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60da06c0-f058-4870-aceb-771126eae974 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:25.693313 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60da06c0-f058-4870-aceb-771126eae974 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:25.694823 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60da06c0-f058-4870-aceb-771126eae974 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:25.695379 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c43dca5-b2d2-4d4e-844b-7da237f0a424 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:25.696012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c43dca5-b2d2-4d4e-844b-7da237f0a424 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:25.696292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c43dca5-b2d2-4d4e-844b-7da237f0a424 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:25.696567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c43dca5-b2d2-4d4e-844b-7da237f0a424 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:25.697687 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c43dca5-b2d2-4d4e-844b-7da237f0a424 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:25.699143 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c43dca5-b2d2-4d4e-844b-7da237f0a424 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:25.700693 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 10/18] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:46:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:46:25.706936 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c43dca5-b2d2-4d4e-844b-7da237f0a424 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:25.714137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-60da06c0-f058-4870-aceb-771126eae974 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:46:25.714137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:25.714137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:25.714137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:25.714137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:25.714137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:25.714137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:25.714137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:25.714137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:25.714137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:25.714137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:25.714137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:25.714137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:46:25.714137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:46:25.714137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:46:25.714137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:25.715342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:46:25.716120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:46:25.716120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:46:25.716120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:46:25.716120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 9/19] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:46:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:46:25.721551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60da06c0-f058-4870-aceb-771126eae974 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:25.778107 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:25.787502 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:25.806696 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c43dca5-b2d2-4d4e-844b-7da237f0a424 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:25.823861 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60da06c0-f058-4870-aceb-771126eae974 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:25.839959 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8e4970-ff84-442b-a50d-f007b6b6ff02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:25.840573 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8e4970-ff84-442b-a50d-f007b6b6ff02 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:25.840858 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8e4970-ff84-442b-a50d-f007b6b6ff02 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:25.841193 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8e4970-ff84-442b-a50d-f007b6b6ff02 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:25.841921 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8e4970-ff84-442b-a50d-f007b6b6ff02 None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.domain.id=default, user.id=c81176b6299a493294b869b497bad7f4) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:25.842265 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8e4970-ff84-442b-a50d-f007b6b6ff02 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:25.843713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8e4970-ff84-442b-a50d-f007b6b6ff02 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:25.851186 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 11/20] 10.222.0.27 () {56 vars in 1322 bytes} [Wed Sep 7 03:46:25 2022] GET /identity/v3/role_assignments?user.id=c81176b6299a493294b869b497bad7f4&scope.domain.id=default&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 233 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:46:25.857699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0204d5-9d70-4baf-ac8a-7e9c7c4a7943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:25.858365 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0204d5-9d70-4baf-ac8a-7e9c7c4a7943 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:25.858647 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0204d5-9d70-4baf-ac8a-7e9c7c4a7943 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:25.858924 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0204d5-9d70-4baf-ac8a-7e9c7c4a7943 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:25.859586 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0204d5-9d70-4baf-ac8a-7e9c7c4a7943 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:25.859892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0204d5-9d70-4baf-ac8a-7e9c7c4a7943 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:25.861378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0204d5-9d70-4baf-ac8a-7e9c7c4a7943 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:25.881265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 10/21] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:46:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:46:28.080945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f60931-a7d4-4d78-9a59-0f2a77ee4a24 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:28.081456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f60931-a7d4-4d78-9a59-0f2a77ee4a24 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:28.081530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f60931-a7d4-4d78-9a59-0f2a77ee4a24 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:28.083017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 12/22] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:28.088175 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a3b593-efcc-4a1e-8fd0-7c4f68099634 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:28.088598 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a3b593-efcc-4a1e-8fd0-7c4f68099634 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:28.088771 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a3b593-efcc-4a1e-8fd0-7c4f68099634 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:28.115239 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-d4a3b593-efcc-4a1e-8fd0-7c4f68099634 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:28.116473 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-d4a3b593-efcc-4a1e-8fd0-7c4f68099634 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:28.201489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 11/23] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:28.705558 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1348b1b8-424d-4743-94e0-9c612998f0a7 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:28.705930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1348b1b8-424d-4743-94e0-9c612998f0a7 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:28.706164 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1348b1b8-424d-4743-94e0-9c612998f0a7 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:28.707477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 13/24] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:46:28.713105 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44800f1-dc83-4a14-a527-9ae91463fc59 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:28.713469 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44800f1-dc83-4a14-a527-9ae91463fc59 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:28.713751 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44800f1-dc83-4a14-a527-9ae91463fc59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:28.740614 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-c44800f1-dc83-4a14-a527-9ae91463fc59 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:28.741865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-c44800f1-dc83-4a14-a527-9ae91463fc59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:28.758067 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1348b1b8-424d-4743-94e0-9c612998f0a7 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:28.856033 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:28.904746 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 12/25] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:28.905832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bf1cd2-1fd1-48cf-807f-d1d182fcc991 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:28.906138 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bf1cd2-1fd1-48cf-807f-d1d182fcc991 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:28.906409 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bf1cd2-1fd1-48cf-807f-d1d182fcc991 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:28.908149 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 13/26] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:28.917300 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2b838b-75a8-41c0-9941-7f183b16e990 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:28.917584 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2b838b-75a8-41c0-9941-7f183b16e990 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:28.917865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2b838b-75a8-41c0-9941-7f183b16e990 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:28.951166 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-ba2b838b-75a8-41c0-9941-7f183b16e990 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:28.952490 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-ba2b838b-75a8-41c0-9941-7f183b16e990 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:29.000735 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1348b1b8-424d-4743-94e0-9c612998f0a7 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:29.062824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 14/27] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:46:29.098499 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabff53a-33bb-49a2-8830-c20975422f2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:29.099650 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d49ed8-2195-4d1b-a5a8-8d9bf6f3d3ec None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:29.100040 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d49ed8-2195-4d1b-a5a8-8d9bf6f3d3ec None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:29.100212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d49ed8-2195-4d1b-a5a8-8d9bf6f3d3ec None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:29.100561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabff53a-33bb-49a2-8830-c20975422f2f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:29.101044 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabff53a-33bb-49a2-8830-c20975422f2f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:29.101511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabff53a-33bb-49a2-8830-c20975422f2f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:29.101618 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 15/28] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:46:29.112501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52b02cd-5c5b-4f89-b7c0-73c9bb8674fb None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:29.112778 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52b02cd-5c5b-4f89-b7c0-73c9bb8674fb None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:29.113051 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52b02cd-5c5b-4f89-b7c0-73c9bb8674fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:29.121251 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabff53a-33bb-49a2-8830-c20975422f2f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:29.127488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-aabff53a-33bb-49a2-8830-c20975422f2f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:46:29.127488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:29.127488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:29.127488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:29.127488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:29.127488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:29.127488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:29.127488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:29.127488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:29.127488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:29.127488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:29.127488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:29.127488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:29.127488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:29.127488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:29.127488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:29.128311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:29.128311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:29.128311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:29.128311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:29.128311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:29.128311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:29.128311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:29.128311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:29.128311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:29.128311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:29.128311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:29.128311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:29.128311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:29.128311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:29.128311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:29.128311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:29.129491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:29.129491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:29.129491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:29.129491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:29.129491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:29.129491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:29.129491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:29.129491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:29.129491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:29.129491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:29.129491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:46:29.129491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:46:29.129491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 14/29] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:46:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 371 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:29.134343 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabff53a-33bb-49a2-8830-c20975422f2f None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:29.142450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:29.146854 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-f52b02cd-5c5b-4f89-b7c0-73c9bb8674fb None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:29.148128 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-f52b02cd-5c5b-4f89-b7c0-73c9bb8674fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:29.299588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 16/30] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:29.300289 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52b02cd-5c5b-4f89-b7c0-73c9bb8674fb None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:29.350357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabff53a-33bb-49a2-8830-c20975422f2f None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:29.369266 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:29.408524 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c1c8cf-1c23-4cc8-8458-8ab3df4dcbfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:29.409145 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c1c8cf-1c23-4cc8-8458-8ab3df4dcbfb None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:29.409424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c1c8cf-1c23-4cc8-8458-8ab3df4dcbfb None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:29.409694 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c1c8cf-1c23-4cc8-8458-8ab3df4dcbfb None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:29.410397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c1c8cf-1c23-4cc8-8458-8ab3df4dcbfb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:29.410700 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c1c8cf-1c23-4cc8-8458-8ab3df4dcbfb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:29.412104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c1c8cf-1c23-4cc8-8458-8ab3df4dcbfb None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:29.436326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 15/31] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:46:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:29.437556 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c1c8cf-1c23-4cc8-8458-8ab3df4dcbfb None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:29.518401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52b02cd-5c5b-4f89-b7c0-73c9bb8674fb None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:29.549335 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:29.591915 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c902315-05d5-45b5-84f2-1a0f9eac9a34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:29.592556 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c902315-05d5-45b5-84f2-1a0f9eac9a34 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:29.592834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c902315-05d5-45b5-84f2-1a0f9eac9a34 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:29.593104 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c902315-05d5-45b5-84f2-1a0f9eac9a34 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:29.613620 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c902315-05d5-45b5-84f2-1a0f9eac9a34 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:29.615514 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c1c8cf-1c23-4cc8-8458-8ab3df4dcbfb None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:29.615942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c902315-05d5-45b5-84f2-1a0f9eac9a34 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:29.633645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-8c902315-05d5-45b5-84f2-1a0f9eac9a34 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:46:29.633645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:29.633645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:29.633645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:29.633645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:29.633645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:29.633645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:29.633645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:29.633645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:29.633645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:29.633645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:29.633645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:29.633645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:46:29.633645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:46:29.633645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:46:29.633645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:29.634591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:46:29.635372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:46:29.635372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:46:29.635372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:46:29.637686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 17/32] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:46:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:29.639022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c902315-05d5-45b5-84f2-1a0f9eac9a34 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:29.651206 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73801f41-83d1-4ab4-8a08-97a9610b13f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:29.651826 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73801f41-83d1-4ab4-8a08-97a9610b13f1 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:29.652106 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73801f41-83d1-4ab4-8a08-97a9610b13f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:29.652379 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73801f41-83d1-4ab4-8a08-97a9610b13f1 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:29.653054 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73801f41-83d1-4ab4-8a08-97a9610b13f1 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:29.654470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73801f41-83d1-4ab4-8a08-97a9610b13f1 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:29.740729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:29.753978 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c902315-05d5-45b5-84f2-1a0f9eac9a34 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:29.790830 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e58d46-d9bb-40dd-8a32-1f2474b908b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:29.791595 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e58d46-d9bb-40dd-8a32-1f2474b908b7 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:29.791972 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e58d46-d9bb-40dd-8a32-1f2474b908b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:29.792349 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e58d46-d9bb-40dd-8a32-1f2474b908b7 None admin] PATH_INFO: `/v3/users/c81176b6299a493294b869b497bad7f4` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:29.792869 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 16/33] 10.222.0.27 () {60 vars in 1131 bytes} [Wed Sep 7 03:46:29 2022] POST /identity/v3/roles => generated 219 bytes in 356 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:46:29.793999 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73801f41-83d1-4ab4-8a08-97a9610b13f1 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:29.794333 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e58d46-d9bb-40dd-8a32-1f2474b908b7 None admin] RBAC: Authorizing `identity:get_user(user_id=c81176b6299a493294b869b497bad7f4)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:29.795965 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e58d46-d9bb-40dd-8a32-1f2474b908b7 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:29.798528 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 18/34] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:29 2022] GET /identity/v3/users/c81176b6299a493294b869b497bad7f4 => generated 247 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:29.800416 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e58d46-d9bb-40dd-8a32-1f2474b908b7 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:29.917094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73801f41-83d1-4ab4-8a08-97a9610b13f1 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:29.923632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e58d46-d9bb-40dd-8a32-1f2474b908b7 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:29.946882 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:29.959983 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7f6f59-4973-4490-b9cb-65cb3e79ff16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:29.961028 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7f6f59-4973-4490-b9cb-65cb3e79ff16 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:29.961449 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7f6f59-4973-4490-b9cb-65cb3e79ff16 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:29.961892 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7f6f59-4973-4490-b9cb-65cb3e79ff16 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:29.962823 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7f6f59-4973-4490-b9cb-65cb3e79ff16 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:29.964772 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7f6f59-4973-4490-b9cb-65cb3e79ff16 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:29.967657 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af40ac1-f8f6-4e12-941d-1f963b35f1c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:29.968424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af40ac1-f8f6-4e12-941d-1f963b35f1c7 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:29.968812 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af40ac1-f8f6-4e12-941d-1f963b35f1c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:29.969191 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af40ac1-f8f6-4e12-941d-1f963b35f1c7 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:29.970561 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af40ac1-f8f6-4e12-941d-1f963b35f1c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:29.970976 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af40ac1-f8f6-4e12-941d-1f963b35f1c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:29.972717 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af40ac1-f8f6-4e12-941d-1f963b35f1c7 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:29.985767 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 17/35] 10.222.0.27 () {60 vars in 1131 bytes} [Wed Sep 7 03:46:29 2022] POST /identity/v3/roles => generated 225 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:46:29.987039 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7f6f59-4973-4490-b9cb-65cb3e79ff16 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:29.996164 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 19/36] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:46:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:29.997326 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af40ac1-f8f6-4e12-941d-1f963b35f1c7 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:30.054809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:30.135851 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7f6f59-4973-4490-b9cb-65cb3e79ff16 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:30.194569 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aacd489-6e02-428e-87a5-3eff591231f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:30.195941 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aacd489-6e02-428e-87a5-3eff591231f4 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:30.196442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aacd489-6e02-428e-87a5-3eff591231f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:30.196897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aacd489-6e02-428e-87a5-3eff591231f4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:30.198707 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aacd489-6e02-428e-87a5-3eff591231f4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:30.200706 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aacd489-6e02-428e-87a5-3eff591231f4 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:30.204019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af40ac1-f8f6-4e12-941d-1f963b35f1c7 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:30.205329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 18/37] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:46:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:30.207231 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be05f8a-2263-4300-9243-18e4ac2b0c76 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:30.207708 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be05f8a-2263-4300-9243-18e4ac2b0c76 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:30.208213 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be05f8a-2263-4300-9243-18e4ac2b0c76 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:30.211497 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 19/38] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:30 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:30.213623 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be05f8a-2263-4300-9243-18e4ac2b0c76 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:30.263486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e783b26-10f4-4a3b-978d-94e64b8a6081 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:30.264266 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e783b26-10f4-4a3b-978d-94e64b8a6081 None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:30.264516 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e783b26-10f4-4a3b-978d-94e64b8a6081 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:30.264857 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e783b26-10f4-4a3b-978d-94e64b8a6081 None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:30.265606 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e783b26-10f4-4a3b-978d-94e64b8a6081 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:30.267941 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e783b26-10f4-4a3b-978d-94e64b8a6081 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:30.300556 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:30.384821 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be05f8a-2263-4300-9243-18e4ac2b0c76 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:30.423962 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 20/39] 10.222.0.27 () {60 vars in 1141 bytes} [Wed Sep 7 03:46:29 2022] POST /identity/v3/projects => generated 301 bytes in 427 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:46:30.425511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8101d995-a28e-43e9-9c08-b293e7f5e19b None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:30.425813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8101d995-a28e-43e9-9c08-b293e7f5e19b None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:30.426096 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8101d995-a28e-43e9-9c08-b293e7f5e19b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:30.435639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83502dd-a3ff-4a5d-8235-a6b3746e259a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:30.436342 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83502dd-a3ff-4a5d-8235-a6b3746e259a None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:30.436623 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83502dd-a3ff-4a5d-8235-a6b3746e259a None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:30.436907 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83502dd-a3ff-4a5d-8235-a6b3746e259a None admin] PATH_INFO: `/v3/domains/default/users/c81176b6299a493294b869b497bad7f4/roles/9d5b19a61c46475d8048b1bd66ca25a1` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:30.440999 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83502dd-a3ff-4a5d-8235-a6b3746e259a None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=c81176b6299a493294b869b497bad7f4, role_id=9d5b19a61c46475d8048b1bd66ca25a1)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:30.442818 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83502dd-a3ff-4a5d-8235-a6b3746e259a None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:30.452181 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-8101d995-a28e-43e9-9c08-b293e7f5e19b None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:30.455080 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-8101d995-a28e-43e9-9c08-b293e7f5e19b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:30.461922 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 20/40] 10.222.0.27 () {58 vars in 1361 bytes} [Wed Sep 7 03:46:30 2022] PUT /identity/v3/domains/default/users/c81176b6299a493294b869b497bad7f4/roles/9d5b19a61c46475d8048b1bd66ca25a1 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:46:30.463243 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d69600-7056-4cac-a31e-8ab5cb816f05 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:30.463590 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d69600-7056-4cac-a31e-8ab5cb816f05 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:30.463877 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d69600-7056-4cac-a31e-8ab5cb816f05 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:30.465795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 21/41] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:30.473199 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cb10d7-8f96-426f-91b8-d890c563dc7e None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:30.473485 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cb10d7-8f96-426f-91b8-d890c563dc7e None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:30.473769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cb10d7-8f96-426f-91b8-d890c563dc7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:30.501930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-a9cb10d7-8f96-426f-91b8-d890c563dc7e None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:30.503754 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-a9cb10d7-8f96-426f-91b8-d890c563dc7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:30.555307 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 21/42] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:30.556052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:30.597915 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 22/43] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:46:30.835483 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8101d995-a28e-43e9-9c08-b293e7f5e19b None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:30.936960 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8101d995-a28e-43e9-9c08-b293e7f5e19b None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:30.942835 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cb10d7-8f96-426f-91b8-d890c563dc7e None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:30.978919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d59f980-f4fd-4ec7-8299-3cc70f036310 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:30.979538 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d59f980-f4fd-4ec7-8299-3cc70f036310 None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:30.979825 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d59f980-f4fd-4ec7-8299-3cc70f036310 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:30.980098 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d59f980-f4fd-4ec7-8299-3cc70f036310 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:30.980767 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d59f980-f4fd-4ec7-8299-3cc70f036310 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:30.982342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d59f980-f4fd-4ec7-8299-3cc70f036310 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:31.000993 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 22/44] 10.222.0.27 () {60 vars in 1131 bytes} [Wed Sep 7 03:46:30 2022] POST /identity/v3/roles => generated 223 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:46:31.003624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:31.071328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cb10d7-8f96-426f-91b8-d890c563dc7e None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:31.112602 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026e85e5-39a0-4ffe-b276-ff3314e9e175 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:31.113169 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026e85e5-39a0-4ffe-b276-ff3314e9e175 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:31.113456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026e85e5-39a0-4ffe-b276-ff3314e9e175 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:31.113732 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026e85e5-39a0-4ffe-b276-ff3314e9e175 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:31.114922 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026e85e5-39a0-4ffe-b276-ff3314e9e175 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:31.116449 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026e85e5-39a0-4ffe-b276-ff3314e9e175 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:31.118278 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 23/45] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:46:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:46:31.123829 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d59f980-f4fd-4ec7-8299-3cc70f036310 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:31.188120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:31.235537 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d59f980-f4fd-4ec7-8299-3cc70f036310 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:31.270928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5150ad12-521f-49de-814a-a39f95be4afb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:31.271554 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5150ad12-521f-49de-814a-a39f95be4afb None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:31.271841 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5150ad12-521f-49de-814a-a39f95be4afb None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:31.272121 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5150ad12-521f-49de-814a-a39f95be4afb None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:31.272794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5150ad12-521f-49de-814a-a39f95be4afb None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:31.274291 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5150ad12-521f-49de-814a-a39f95be4afb None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:31.302874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 23/46] 10.222.0.27 () {60 vars in 1141 bytes} [Wed Sep 7 03:46:31 2022] POST /identity/v3/projects => generated 312 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:46:32.710388 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291f0f87-3d5e-4826-8b9a-513fc5ddeb04 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:32.710935 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291f0f87-3d5e-4826-8b9a-513fc5ddeb04 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:32.710935 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291f0f87-3d5e-4826-8b9a-513fc5ddeb04 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:32.712292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 24/47] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:32.717680 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b8fc4e-810d-428c-b8d8-3e031c356c4a None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:32.718109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b8fc4e-810d-428c-b8d8-3e031c356c4a None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:32.718382 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b8fc4e-810d-428c-b8d8-3e031c356c4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:32.736875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-15b8fc4e-810d-428c-b8d8-3e031c356c4a None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:32.738051 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-15b8fc4e-810d-428c-b8d8-3e031c356c4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:32.803025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 24/48] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:33.068250 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291f0f87-3d5e-4826-8b9a-513fc5ddeb04 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:33.276574 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291f0f87-3d5e-4826-8b9a-513fc5ddeb04 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:33.289658 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:33.311500 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246c02be-cbc9-4c5c-a2a2-86df748300b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:33.312141 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246c02be-cbc9-4c5c-a2a2-86df748300b9 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:33.312424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246c02be-cbc9-4c5c-a2a2-86df748300b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:33.312702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246c02be-cbc9-4c5c-a2a2-86df748300b9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:33.323409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246c02be-cbc9-4c5c-a2a2-86df748300b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:33.328663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-246c02be-cbc9-4c5c-a2a2-86df748300b9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:46:33.328663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:33.328663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:33.328663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:33.328663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:33.328663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:33.328663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:33.328663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:33.328663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:33.328663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:33.328663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:33.328663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:33.328663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:33.328663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:33.328663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:33.328663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:33.329575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:33.329575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:33.329575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:33.329575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:33.329575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:33.329575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:33.329575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:33.329575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:33.329575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:33.329575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:33.329575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:33.329575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:33.329575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:33.329575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:33.329575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:33.329575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:33.330338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:33.330338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:33.330338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:33.330338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:33.330338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:33.330338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:33.330338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:33.330338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:33.330338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:33.330338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:33.330338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:46:33.330338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:46:33.330338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 25/49] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:46:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:33.336296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b8fc4e-810d-428c-b8d8-3e031c356c4a None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:33.381733 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:33.440094 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b8fc4e-810d-428c-b8d8-3e031c356c4a None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:33.457606 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03db32f-e0aa-4953-a57e-38b99ab27419 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:33.458065 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03db32f-e0aa-4953-a57e-38b99ab27419 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:33.458433 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03db32f-e0aa-4953-a57e-38b99ab27419 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:33.459769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 26/50] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:33.465090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ecb30b-c536-4185-9e0a-fe4570e86d99 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:33.465381 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ecb30b-c536-4185-9e0a-fe4570e86d99 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:33.465663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ecb30b-c536-4185-9e0a-fe4570e86d99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:33.474240 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f5160c-ea1a-44bd-ac0c-0e4590ef868e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:33.474857 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f5160c-ea1a-44bd-ac0c-0e4590ef868e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:33.475137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f5160c-ea1a-44bd-ac0c-0e4590ef868e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:33.475413 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f5160c-ea1a-44bd-ac0c-0e4590ef868e None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:33.476083 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f5160c-ea1a-44bd-ac0c-0e4590ef868e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:33.476389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f5160c-ea1a-44bd-ac0c-0e4590ef868e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:33.477762 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f5160c-ea1a-44bd-ac0c-0e4590ef868e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:33.483385 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-92ecb30b-c536-4185-9e0a-fe4570e86d99 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:33.484588 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-92ecb30b-c536-4185-9e0a-fe4570e86d99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:33.491167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 25/51] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:46:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:33.496739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f5160c-ea1a-44bd-ac0c-0e4590ef868e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:33.556871 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 27/52] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:46:33.699158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f5160c-ea1a-44bd-ac0c-0e4590ef868e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:33.700569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:33.733871 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b96cd9f-0cab-4acc-aa92-c83181977a71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:33.734568 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b96cd9f-0cab-4acc-aa92-c83181977a71 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:33.734800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b96cd9f-0cab-4acc-aa92-c83181977a71 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:33.735732 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b96cd9f-0cab-4acc-aa92-c83181977a71 None admin] PATH_INFO: `/v3/users/c81176b6299a493294b869b497bad7f4` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:33.736886 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b96cd9f-0cab-4acc-aa92-c83181977a71 None admin] RBAC: Authorizing `identity:get_user(user_id=c81176b6299a493294b869b497bad7f4)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:33.738349 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b96cd9f-0cab-4acc-aa92-c83181977a71 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:33.739938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 26/53] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:33 2022] GET /identity/v3/users/c81176b6299a493294b869b497bad7f4 => generated 247 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:46:33.745447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ecb30b-c536-4185-9e0a-fe4570e86d99 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:33.826139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b96cd9f-0cab-4acc-aa92-c83181977a71 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:33.842660 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ecb30b-c536-4185-9e0a-fe4570e86d99 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:33.844169 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:33.878893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a12d20-deeb-487e-bb27-46972bdc8ff8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:33.879559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a12d20-deeb-487e-bb27-46972bdc8ff8 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:33.879843 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a12d20-deeb-487e-bb27-46972bdc8ff8 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:33.880118 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a12d20-deeb-487e-bb27-46972bdc8ff8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:33.881228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a12d20-deeb-487e-bb27-46972bdc8ff8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:33.882711 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a12d20-deeb-487e-bb27-46972bdc8ff8 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:33.884353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 28/54] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:46:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:46:33.890018 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a12d20-deeb-487e-bb27-46972bdc8ff8 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:33.925368 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b96cd9f-0cab-4acc-aa92-c83181977a71 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:33.961088 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f370a33-18d2-409d-a960-28b04be950c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:33.961650 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f370a33-18d2-409d-a960-28b04be950c6 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:33.961931 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f370a33-18d2-409d-a960-28b04be950c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:33.962239 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f370a33-18d2-409d-a960-28b04be950c6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:33.963348 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f370a33-18d2-409d-a960-28b04be950c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:33.964784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f370a33-18d2-409d-a960-28b04be950c6 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:33.966390 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 27/55] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:46:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:46:33.971818 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f370a33-18d2-409d-a960-28b04be950c6 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:33.991561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a12d20-deeb-487e-bb27-46972bdc8ff8 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:34.010921 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:34.025352 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87831dd2-f96e-4662-ba07-de095059f19c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:34.025971 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87831dd2-f96e-4662-ba07-de095059f19c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:34.026288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87831dd2-f96e-4662-ba07-de095059f19c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:34.026564 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87831dd2-f96e-4662-ba07-de095059f19c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:34.027291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87831dd2-f96e-4662-ba07-de095059f19c None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.domain.id=default, user.id=c81176b6299a493294b869b497bad7f4) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:34.027600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87831dd2-f96e-4662-ba07-de095059f19c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:34.029051 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87831dd2-f96e-4662-ba07-de095059f19c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:34.034827 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 29/56] 10.222.0.27 () {56 vars in 1322 bytes} [Wed Sep 7 03:46:33 2022] GET /identity/v3/role_assignments?user.id=c81176b6299a493294b869b497bad7f4&scope.domain.id=default&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 530 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:34.075453 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f370a33-18d2-409d-a960-28b04be950c6 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:34.111739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d41ac6-1eab-41b2-9cde-e96d7f1a6774 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:34.112370 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d41ac6-1eab-41b2-9cde-e96d7f1a6774 None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:34.112655 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d41ac6-1eab-41b2-9cde-e96d7f1a6774 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:34.112934 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d41ac6-1eab-41b2-9cde-e96d7f1a6774 None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:34.113604 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d41ac6-1eab-41b2-9cde-e96d7f1a6774 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:34.115103 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d41ac6-1eab-41b2-9cde-e96d7f1a6774 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:34.143824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 28/57] 10.222.0.27 () {60 vars in 1140 bytes} [Wed Sep 7 03:46:33 2022] POST /identity/v3/projects => generated 298 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 07 03:46:36.277579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25eeb01-cd47-4fb2-8e54-43f5cc2c21a4 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:36.277983 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25eeb01-cd47-4fb2-8e54-43f5cc2c21a4 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:36.278160 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25eeb01-cd47-4fb2-8e54-43f5cc2c21a4 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:36.279501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 30/58] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:36.285316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3111c01f-1937-400d-99e0-e921e01915cc None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:36.285726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3111c01f-1937-400d-99e0-e921e01915cc None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:36.285873 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3111c01f-1937-400d-99e0-e921e01915cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:36.303639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-3111c01f-1937-400d-99e0-e921e01915cc None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:36.304849 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-3111c01f-1937-400d-99e0-e921e01915cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:36.369058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 29/59] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:36.640940 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25eeb01-cd47-4fb2-8e54-43f5cc2c21a4 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:36.675768 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:36.740226 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25eeb01-cd47-4fb2-8e54-43f5cc2c21a4 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:36.773541 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0497f6-1155-4015-aa93-d7729ca32abd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:36.774186 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0497f6-1155-4015-aa93-d7729ca32abd None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:36.774468 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0497f6-1155-4015-aa93-d7729ca32abd None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:36.774741 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0497f6-1155-4015-aa93-d7729ca32abd None admin] PATH_INFO: `/v3/domains/default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:36.775838 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0497f6-1155-4015-aa93-d7729ca32abd None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:36.777278 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0497f6-1155-4015-aa93-d7729ca32abd None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:36.778844 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 31/60] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:46:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:36.784349 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3111c01f-1937-400d-99e0-e921e01915cc None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:36.804821 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:36.884037 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3111c01f-1937-400d-99e0-e921e01915cc None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:36.916879 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4477ad20-15f6-4ed3-b06d-cbd464ec46d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:36.917428 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4477ad20-15f6-4ed3-b06d-cbd464ec46d1 None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:36.917713 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4477ad20-15f6-4ed3-b06d-cbd464ec46d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:36.917986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4477ad20-15f6-4ed3-b06d-cbd464ec46d1 None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:36.918657 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4477ad20-15f6-4ed3-b06d-cbd464ec46d1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:36.920098 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4477ad20-15f6-4ed3-b06d-cbd464ec46d1 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:36.947365 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 30/61] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:46:36 2022] POST /identity/v3/users => generated 275 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:46:40.276079 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0cff04-8b0a-487c-bd26-217e11c89a95 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:40.277011 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0cff04-8b0a-487c-bd26-217e11c89a95 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:40.277517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0cff04-8b0a-487c-bd26-217e11c89a95 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:40.279317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 32/62] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:40.285120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8dd655-abab-474c-b08b-8b1ba5a7f0af None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:40.285918 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8dd655-abab-474c-b08b-8b1ba5a7f0af None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:40.286435 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8dd655-abab-474c-b08b-8b1ba5a7f0af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:40.315162 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-9e8dd655-abab-474c-b08b-8b1ba5a7f0af None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:40.316750 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-9e8dd655-abab-474c-b08b-8b1ba5a7f0af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:40.340013 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b8a1c7-97ea-4504-80e4-f6ed58039203 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:40.340612 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b8a1c7-97ea-4504-80e4-f6ed58039203 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:40.341107 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b8a1c7-97ea-4504-80e4-f6ed58039203 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:40.342920 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 33/63] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:40.350305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b97a4cb-29b4-4609-a828-2a793412f39b None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:40.350843 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b97a4cb-29b4-4609-a828-2a793412f39b None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:40.351317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b97a4cb-29b4-4609-a828-2a793412f39b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:40.384063 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-0b97a4cb-29b4-4609-a828-2a793412f39b None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:40.387617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-0b97a4cb-29b4-4609-a828-2a793412f39b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:40.414737 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 31/64] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:46:40.484455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 34/65] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:46:40.657732 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb9e530-e5e5-4a37-9245-4aacd1ed9998 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:40.658422 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb9e530-e5e5-4a37-9245-4aacd1ed9998 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:40.658900 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb9e530-e5e5-4a37-9245-4aacd1ed9998 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:40.660745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 32/66] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:40.666719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa1ecce-812c-4adc-bbf5-36ff1169e558 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:40.667269 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa1ecce-812c-4adc-bbf5-36ff1169e558 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:40.667748 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa1ecce-812c-4adc-bbf5-36ff1169e558 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:40.690663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-6fa1ecce-812c-4adc-bbf5-36ff1169e558 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:40.692208 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-6fa1ecce-812c-4adc-bbf5-36ff1169e558 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:40.727533 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:40.781792 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd422e93-aeed-46b7-98fd-e8deffea9039 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:40.782736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd422e93-aeed-46b7-98fd-e8deffea9039 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:40.783316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd422e93-aeed-46b7-98fd-e8deffea9039 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:40.784857 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 33/67] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:46:40.787180 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 35/68] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:40.790306 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7773d877-4030-4895-922a-8866149a6250 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:40.790838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7773d877-4030-4895-922a-8866149a6250 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:40.791328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7773d877-4030-4895-922a-8866149a6250 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:40.813531 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:40.819057 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa1ecce-812c-4adc-bbf5-36ff1169e558 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:40.834945 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-7773d877-4030-4895-922a-8866149a6250 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:40.836187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-7773d877-4030-4895-922a-8866149a6250 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:40.923900 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 34/69] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:46:40.924692 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7773d877-4030-4895-922a-8866149a6250 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:40.951827 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa1ecce-812c-4adc-bbf5-36ff1169e558 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:41.015522 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683bc244-06e7-4148-b475-b70b06bbf133 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:41.016995 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683bc244-06e7-4148-b475-b70b06bbf133 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:41.017424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683bc244-06e7-4148-b475-b70b06bbf133 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:41.017840 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683bc244-06e7-4148-b475-b70b06bbf133 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:41.029365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683bc244-06e7-4148-b475-b70b06bbf133 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:41.034946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-683bc244-06e7-4148-b475-b70b06bbf133 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:46:41.034946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:41.034946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:41.034946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:41.034946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:41.034946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:41.034946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:41.034946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:41.034946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:41.034946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:41.034946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:41.034946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:41.034946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:41.034946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:41.034946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:41.034946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:41.035997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:41.035997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:41.035997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:41.035997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:41.035997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:41.035997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:41.035997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:41.035997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:41.035997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:41.035997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:41.035997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:41.035997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:41.035997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:41.035997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:41.035997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:41.035997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:41.036743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:41.036743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:41.036743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:41.036743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:41.036743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:41.036743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:41.036743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:41.036743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:41.036743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:41.036743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:41.036743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:46:41.036743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:46:41.037312 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7773d877-4030-4895-922a-8866149a6250 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:41.039124 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 36/70] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:46:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:41.041151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e332654f-2b80-4f9f-81e7-4e42b3543a99 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:41.041655 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e332654f-2b80-4f9f-81e7-4e42b3543a99 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:41.042168 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e332654f-2b80-4f9f-81e7-4e42b3543a99 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:41.043635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 37/71] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:41.050027 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e332654f-2b80-4f9f-81e7-4e42b3543a99 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:41.074042 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7748f1-5f7b-4622-8046-44e825092719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:41.075067 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7748f1-5f7b-4622-8046-44e825092719 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:41.075652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7748f1-5f7b-4622-8046-44e825092719 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:41.076139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7748f1-5f7b-4622-8046-44e825092719 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:41.080495 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:41.081930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:41.089787 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7748f1-5f7b-4622-8046-44e825092719 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:41.096443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-4d7748f1-5f7b-4622-8046-44e825092719 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:46:41.096443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:41.096443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:41.096443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:41.096443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:41.096443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:41.096443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:41.096443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:41.096443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:41.096443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:41.096443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:41.096443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:41.096443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:41.096443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:41.096443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:41.096443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:41.097324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:41.097324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:41.097324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:41.097324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:41.097324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:41.097324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:41.097324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:41.097324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:41.097324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:41.097324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:41.097324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:41.097324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:41.097324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:41.097324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:41.097324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:41.097324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:41.098966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:41.098966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:41.098966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:41.098966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:41.098966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:41.098966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:41.098966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:41.098966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:41.098966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:41.098966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:41.098966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:46:41.098966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:46:41.098966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 35/72] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:46:40 2022] GET /identity/v3/roles/member => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:41.098966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2389ffb1-ba9b-45e2-bc03-fdf226da74d3 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:41.099644 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2389ffb1-ba9b-45e2-bc03-fdf226da74d3 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:41.099644 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2389ffb1-ba9b-45e2-bc03-fdf226da74d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:41.117816 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-2389ffb1-ba9b-45e2-bc03-fdf226da74d3 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:41.119052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-2389ffb1-ba9b-45e2-bc03-fdf226da74d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:41.173596 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e332654f-2b80-4f9f-81e7-4e42b3543a99 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:41.193809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 36/73] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:41.195480 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2389ffb1-ba9b-45e2-bc03-fdf226da74d3 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:41.209385 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd164ff9-e04f-48fb-8782-8b895089c8e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:41.210015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd164ff9-e04f-48fb-8782-8b895089c8e0 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:41.210332 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd164ff9-e04f-48fb-8782-8b895089c8e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:41.210611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd164ff9-e04f-48fb-8782-8b895089c8e0 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:41.211285 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd164ff9-e04f-48fb-8782-8b895089c8e0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:41.211595 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd164ff9-e04f-48fb-8782-8b895089c8e0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:41.212983 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd164ff9-e04f-48fb-8782-8b895089c8e0 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:41.223919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 38/74] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:46:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:41.224580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd164ff9-e04f-48fb-8782-8b895089c8e0 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:41.309557 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2389ffb1-ba9b-45e2-bc03-fdf226da74d3 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:41.328042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:41.336553 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:41.346438 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd164ff9-e04f-48fb-8782-8b895089c8e0 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:41.346565 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80839c01-b04a-4047-aa9d-6cf1037d5081 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:41.347135 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80839c01-b04a-4047-aa9d-6cf1037d5081 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:41.347417 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80839c01-b04a-4047-aa9d-6cf1037d5081 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:41.347691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80839c01-b04a-4047-aa9d-6cf1037d5081 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:41.358265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80839c01-b04a-4047-aa9d-6cf1037d5081 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:41.364215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-80839c01-b04a-4047-aa9d-6cf1037d5081 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:46:41.364215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:41.364215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:41.364215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:41.364215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:41.364215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:41.364215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:41.364215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:41.364215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:41.364215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:41.364215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:41.364215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:41.364215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:41.364215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:41.364215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:41.364215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:41.365044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:41.365044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:41.365044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:41.365044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:41.365044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:41.365044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:41.365044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:41.365044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:41.365044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:41.365044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:41.365044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:41.365044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:41.365044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:41.365044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:41.365044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:41.365044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:41.366954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:41.366954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:41.366954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:41.366954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:41.366954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:41.366954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:41.366954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:41.366954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:41.366954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:41.366954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:41.366954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:46:41.366954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:46:41.366954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 37/75] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:46:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:41.367642 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80839c01-b04a-4047-aa9d-6cf1037d5081 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:41.388541 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1950b941-9141-49f9-9497-94040a51076b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:41.389496 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1950b941-9141-49f9-9497-94040a51076b None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:41.390007 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1950b941-9141-49f9-9497-94040a51076b None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:41.390399 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1950b941-9141-49f9-9497-94040a51076b None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:41.391293 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1950b941-9141-49f9-9497-94040a51076b None admin] RBAC: Adding query filter params (name=member) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:41.391865 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1950b941-9141-49f9-9497-94040a51076b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:41.393555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1950b941-9141-49f9-9497-94040a51076b None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:41.403727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 39/76] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:46:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:41.404824 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1950b941-9141-49f9-9497-94040a51076b None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:41.471697 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80839c01-b04a-4047-aa9d-6cf1037d5081 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:41.506693 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49b6dec-09d4-4c48-aabd-a6b9fcac8c77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:41.507327 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49b6dec-09d4-4c48-aabd-a6b9fcac8c77 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:41.507450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1950b941-9141-49f9-9497-94040a51076b None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:41.507609 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49b6dec-09d4-4c48-aabd-a6b9fcac8c77 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:41.507885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49b6dec-09d4-4c48-aabd-a6b9fcac8c77 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:41.518519 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49b6dec-09d4-4c48-aabd-a6b9fcac8c77 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:41.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-b49b6dec-09d4-4c48-aabd-a6b9fcac8c77 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:46:41.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:41.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:41.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:41.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:41.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:41.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:41.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:41.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:41.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:41.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:41.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:41.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:41.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:41.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:41.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:41.524288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:41.524288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:41.524288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:41.524288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:41.524288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:41.524288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:41.524288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:41.524288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:41.524288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:41.524288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:41.524288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:41.524288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:41.524288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:41.524288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:41.524288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:41.524288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:41.525048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:41.525048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:41.525048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:41.525048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:41.525048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:41.525048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:41.525048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:41.525048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:41.525048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:41.525048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:41.525048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:46:41.525048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:46:41.525048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 38/77] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:46:41 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:41.525680 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49b6dec-09d4-4c48-aabd-a6b9fcac8c77 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:41.544160 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258971ed-7610-4b76-bfda-2e1c9d68ea2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:41.544798 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258971ed-7610-4b76-bfda-2e1c9d68ea2f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:41.545080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258971ed-7610-4b76-bfda-2e1c9d68ea2f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:41.545357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258971ed-7610-4b76-bfda-2e1c9d68ea2f None admin] PATH_INFO: `/v3/users/c81176b6299a493294b869b497bad7f4` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:41.546557 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258971ed-7610-4b76-bfda-2e1c9d68ea2f None admin] RBAC: Authorizing `identity:get_user(user_id=c81176b6299a493294b869b497bad7f4)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:41.548095 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258971ed-7610-4b76-bfda-2e1c9d68ea2f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:41.549686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 40/78] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:41 2022] GET /identity/v3/users/c81176b6299a493294b869b497bad7f4 => generated 247 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:41.550471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258971ed-7610-4b76-bfda-2e1c9d68ea2f None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:41.565872 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:41.574141 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:41.628180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49b6dec-09d4-4c48-aabd-a6b9fcac8c77 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:41.651663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258971ed-7610-4b76-bfda-2e1c9d68ea2f None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:41.661825 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2265a2-814c-4e1e-926d-5983e1f8886e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:41.662459 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2265a2-814c-4e1e-926d-5983e1f8886e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:41.662743 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2265a2-814c-4e1e-926d-5983e1f8886e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:41.663022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2265a2-814c-4e1e-926d-5983e1f8886e None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:41.663687 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2265a2-814c-4e1e-926d-5983e1f8886e None admin] RBAC: Adding query filter params (name=member) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:41.663993 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2265a2-814c-4e1e-926d-5983e1f8886e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:41.665382 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2265a2-814c-4e1e-926d-5983e1f8886e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:41.674026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 39/79] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:46:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:41.674850 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2265a2-814c-4e1e-926d-5983e1f8886e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:41.690778 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37da686-5b6e-41b2-b234-cc813ab7f26a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:41.691425 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37da686-5b6e-41b2-b234-cc813ab7f26a None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:41.691711 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37da686-5b6e-41b2-b234-cc813ab7f26a None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:41.691991 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37da686-5b6e-41b2-b234-cc813ab7f26a None admin] PATH_INFO: `/v3/users/65508fe717ca4eb996135023e2296dd1` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:41.706182 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37da686-5b6e-41b2-b234-cc813ab7f26a None admin] RBAC: Authorizing `identity:get_user(user_id=65508fe717ca4eb996135023e2296dd1)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:41.707647 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37da686-5b6e-41b2-b234-cc813ab7f26a None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:41.709269 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 41/80] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:41 2022] GET /identity/v3/users/65508fe717ca4eb996135023e2296dd1 => generated 275 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:41.710048 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37da686-5b6e-41b2-b234-cc813ab7f26a None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:41.774047 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2265a2-814c-4e1e-926d-5983e1f8886e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:41.781723 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:41.783046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:41.810054 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8870529-b871-4a68-b0b6-e38001428521 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:41.810715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8870529-b871-4a68-b0b6-e38001428521 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:41.811004 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8870529-b871-4a68-b0b6-e38001428521 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:41.811225 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37da686-5b6e-41b2-b234-cc813ab7f26a None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:41.811316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8870529-b871-4a68-b0b6-e38001428521 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:41.812413 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8870529-b871-4a68-b0b6-e38001428521 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:41.814337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8870529-b871-4a68-b0b6-e38001428521 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:41.816025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 40/81] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:46:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:41.816025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8870529-b871-4a68-b0b6-e38001428521 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:41.850470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a409196-8d91-4c5f-a3f2-711e653bacdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:41.851091 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a409196-8d91-4c5f-a3f2-711e653bacdb None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:41.851376 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a409196-8d91-4c5f-a3f2-711e653bacdb None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:41.851653 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a409196-8d91-4c5f-a3f2-711e653bacdb None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:41.852319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a409196-8d91-4c5f-a3f2-711e653bacdb None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:41.852623 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a409196-8d91-4c5f-a3f2-711e653bacdb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:41.854009 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a409196-8d91-4c5f-a3f2-711e653bacdb None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:41.862152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 42/82] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:46:41 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:41.862957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a409196-8d91-4c5f-a3f2-711e653bacdb None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:41.924990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8870529-b871-4a68-b0b6-e38001428521 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:41.959328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e974b76a-089e-4426-97c7-bbd78280b251 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:41.959972 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e974b76a-089e-4426-97c7-bbd78280b251 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:41.960262 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e974b76a-089e-4426-97c7-bbd78280b251 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:41.960539 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e974b76a-089e-4426-97c7-bbd78280b251 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:41.963131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a409196-8d91-4c5f-a3f2-711e653bacdb None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:41.996438 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3963afec-974d-45e9-91b4-de6727be1b8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:41.997071 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3963afec-974d-45e9-91b4-de6727be1b8d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:41.997359 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3963afec-974d-45e9-91b4-de6727be1b8d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:41.997636 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3963afec-974d-45e9-91b4-de6727be1b8d None admin] PATH_INFO: `/v3/users/65508fe717ca4eb996135023e2296dd1` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:41.998798 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3963afec-974d-45e9-91b4-de6727be1b8d None admin] RBAC: Authorizing `identity:get_user(user_id=65508fe717ca4eb996135023e2296dd1)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:42.000799 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3963afec-974d-45e9-91b4-de6727be1b8d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:42.002432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 43/83] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:41 2022] GET /identity/v3/users/65508fe717ca4eb996135023e2296dd1 => generated 275 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:42.003205 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3963afec-974d-45e9-91b4-de6727be1b8d None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:42.005817 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:42.010956 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:42.035109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e974b76a-089e-4426-97c7-bbd78280b251 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:42.036521 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e974b76a-089e-4426-97c7-bbd78280b251 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:42.105662 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3963afec-974d-45e9-91b4-de6727be1b8d None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:42.134017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 41/84] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:46:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:46:42.134879 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e974b76a-089e-4426-97c7-bbd78280b251 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:42.143680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13759a2-2216-45d1-8306-75ad6eaa2a2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:42.144295 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13759a2-2216-45d1-8306-75ad6eaa2a2e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:42.144578 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13759a2-2216-45d1-8306-75ad6eaa2a2e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:42.144853 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13759a2-2216-45d1-8306-75ad6eaa2a2e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:42.211343 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:42.212312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13759a2-2216-45d1-8306-75ad6eaa2a2e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:42.214133 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13759a2-2216-45d1-8306-75ad6eaa2a2e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:42.215981 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:42.235765 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e974b76a-089e-4426-97c7-bbd78280b251 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:42.272023 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80e98bd-cbff-466d-ab23-83c06fe821bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:42.272649 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80e98bd-cbff-466d-ab23-83c06fe821bc None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:42.272935 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80e98bd-cbff-466d-ab23-83c06fe821bc None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:42.273219 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80e98bd-cbff-466d-ab23-83c06fe821bc None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:42.273904 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80e98bd-cbff-466d-ab23-83c06fe821bc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:42.275404 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80e98bd-cbff-466d-ab23-83c06fe821bc None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:42.295991 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 42/85] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:46:42 2022] POST /identity/v3/users => generated 289 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:46:42.297504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80e98bd-cbff-466d-ab23-83c06fe821bc None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:42.323852 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 44/86] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:46:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:46:42.325104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13759a2-2216-45d1-8306-75ad6eaa2a2e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:42.428550 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80e98bd-cbff-466d-ab23-83c06fe821bc None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:42.436096 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13759a2-2216-45d1-8306-75ad6eaa2a2e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:42.438579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:42.461888 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:42.470905 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a997ddd-7fa8-4981-9968-a193556c2764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:42.471635 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a997ddd-7fa8-4981-9968-a193556c2764 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:42.471994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a997ddd-7fa8-4981-9968-a193556c2764 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:42.472278 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a997ddd-7fa8-4981-9968-a193556c2764 None admin] PATH_INFO: `/v3/users/65508fe717ca4eb996135023e2296dd1` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:42.473611 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a997ddd-7fa8-4981-9968-a193556c2764 None admin] RBAC: Authorizing `identity:get_user(user_id=65508fe717ca4eb996135023e2296dd1)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:42.475933 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a997ddd-7fa8-4981-9968-a193556c2764 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:42.477132 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 43/87] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:42 2022] GET /identity/v3/users/65508fe717ca4eb996135023e2296dd1 => generated 275 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:42.477902 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a997ddd-7fa8-4981-9968-a193556c2764 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:42.482087 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12319093-f86b-4aae-8e29-febcd53ae1aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:42.482994 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12319093-f86b-4aae-8e29-febcd53ae1aa None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:42.482994 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12319093-f86b-4aae-8e29-febcd53ae1aa None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:42.483274 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12319093-f86b-4aae-8e29-febcd53ae1aa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:42.563842 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12319093-f86b-4aae-8e29-febcd53ae1aa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:42.565548 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12319093-f86b-4aae-8e29-febcd53ae1aa None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:42.606143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a997ddd-7fa8-4981-9968-a193556c2764 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:42.655773 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b998f14-1f8a-4c1f-8f0f-9103452190a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:42.656985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b998f14-1f8a-4c1f-8f0f-9103452190a0 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:42.658046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b998f14-1f8a-4c1f-8f0f-9103452190a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:42.658395 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b998f14-1f8a-4c1f-8f0f-9103452190a0 None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:42.660052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b998f14-1f8a-4c1f-8f0f-9103452190a0 None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:42.661966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b998f14-1f8a-4c1f-8f0f-9103452190a0 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:42.664449 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 44/88] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:42 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:42.665315 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b998f14-1f8a-4c1f-8f0f-9103452190a0 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:42.693156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 45/89] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:46:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:46:42.694020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12319093-f86b-4aae-8e29-febcd53ae1aa None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:42.696512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:42.759774 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:42.800592 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b998f14-1f8a-4c1f-8f0f-9103452190a0 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:42.818718 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12319093-f86b-4aae-8e29-febcd53ae1aa None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:42.849124 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f77b314-6a5e-4e62-be60-eee6780da9bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:42.849772 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f77b314-6a5e-4e62-be60-eee6780da9bc None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:42.850331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f77b314-6a5e-4e62-be60-eee6780da9bc None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:42.850644 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f77b314-6a5e-4e62-be60-eee6780da9bc None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:42.852508 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f77b314-6a5e-4e62-be60-eee6780da9bc None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:42.853971 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f77b314-6a5e-4e62-be60-eee6780da9bc None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:42.856087 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 45/90] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:42 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:42.856874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f77b314-6a5e-4e62-be60-eee6780da9bc None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:42.858670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee334850-53a5-468f-a2de-2730acc92065 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:42.859288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee334850-53a5-468f-a2de-2730acc92065 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:42.859646 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee334850-53a5-468f-a2de-2730acc92065 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:42.860081 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee334850-53a5-468f-a2de-2730acc92065 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:42.945729 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee334850-53a5-468f-a2de-2730acc92065 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:42.947263 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee334850-53a5-468f-a2de-2730acc92065 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:42.949149 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:42.967208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f77b314-6a5e-4e62-be60-eee6780da9bc None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:43.003239 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3c51c7-c0c1-43de-bb1e-56c312e69299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:43.003860 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3c51c7-c0c1-43de-bb1e-56c312e69299 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:43.004140 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3c51c7-c0c1-43de-bb1e-56c312e69299 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:43.004419 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3c51c7-c0c1-43de-bb1e-56c312e69299 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:43.005149 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3c51c7-c0c1-43de-bb1e-56c312e69299 None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=d6715600d626423b8e8a06d45bb53d08, user.id=c81176b6299a493294b869b497bad7f4) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:43.005466 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3c51c7-c0c1-43de-bb1e-56c312e69299 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:43.006969 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3c51c7-c0c1-43de-bb1e-56c312e69299 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:43.013092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:43.017871 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 46/91] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:46:42 2022] GET /identity/v3/role_assignments?user.id=c81176b6299a493294b869b497bad7f4&scope.project.id=d6715600d626423b8e8a06d45bb53d08&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 259 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:43.018956 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3c51c7-c0c1-43de-bb1e-56c312e69299 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:43.077785 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 46/92] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:46:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:46:43.079148 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee334850-53a5-468f-a2de-2730acc92065 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:43.164862 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3c51c7-c0c1-43de-bb1e-56c312e69299 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:43.203023 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee334850-53a5-468f-a2de-2730acc92065 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:43.205639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:43.205755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4e693c-8947-4dbc-951e-913e57b159b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:43.207036 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4e693c-8947-4dbc-951e-913e57b159b2 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:43.207421 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4e693c-8947-4dbc-951e-913e57b159b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:43.207877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4e693c-8947-4dbc-951e-913e57b159b2 None admin] PATH_INFO: `/v3/projects/e6d05dd0b455478c8762a69b412f4a4a` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:43.209436 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4e693c-8947-4dbc-951e-913e57b159b2 None admin] RBAC: Authorizing `identity:get_project(project_id=e6d05dd0b455478c8762a69b412f4a4a)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:43.211510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4e693c-8947-4dbc-951e-913e57b159b2 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:43.213896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 47/93] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:43 2022] GET /identity/v3/projects/e6d05dd0b455478c8762a69b412f4a4a => generated 312 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:43.214917 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4e693c-8947-4dbc-951e-913e57b159b2 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:43.256862 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4600f3-3c7a-4e5d-8073-d46e09c7ed39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:43.257501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4600f3-3c7a-4e5d-8073-d46e09c7ed39 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:43.257787 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4600f3-3c7a-4e5d-8073-d46e09c7ed39 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:43.258071 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4600f3-3c7a-4e5d-8073-d46e09c7ed39 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:43.258865 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4600f3-3c7a-4e5d-8073-d46e09c7ed39 None admin] RBAC: Adding query filter params (role.id=34235948f8f943a99328516c2171cfff, scope.project.id=d6715600d626423b8e8a06d45bb53d08, user.id=65508fe717ca4eb996135023e2296dd1) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:43.259189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4600f3-3c7a-4e5d-8073-d46e09c7ed39 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:43.260678 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4600f3-3c7a-4e5d-8073-d46e09c7ed39 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:43.274176 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 47/94] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:46:43 2022] GET /identity/v3/role_assignments?user.id=65508fe717ca4eb996135023e2296dd1&scope.project.id=d6715600d626423b8e8a06d45bb53d08&role.id=34235948f8f943a99328516c2171cfff => generated 259 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:43.274176 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4600f3-3c7a-4e5d-8073-d46e09c7ed39 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:43.340328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:43.399495 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4e693c-8947-4dbc-951e-913e57b159b2 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:43.410519 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4600f3-3c7a-4e5d-8073-d46e09c7ed39 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:43.454044 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9152687b-4482-4401-8dcc-9f8e3acee7a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:43.456303 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9152687b-4482-4401-8dcc-9f8e3acee7a0 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:43.456759 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9152687b-4482-4401-8dcc-9f8e3acee7a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:43.457215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9152687b-4482-4401-8dcc-9f8e3acee7a0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:43.458360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9152687b-4482-4401-8dcc-9f8e3acee7a0 None admin] RBAC: Adding query filter params (role.id=34235948f8f943a99328516c2171cfff, scope.project.id=e6d05dd0b455478c8762a69b412f4a4a, user.id=65508fe717ca4eb996135023e2296dd1) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:43.458965 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9152687b-4482-4401-8dcc-9f8e3acee7a0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:43.460879 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9152687b-4482-4401-8dcc-9f8e3acee7a0 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:43.468700 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 48/95] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:46:43 2022] GET /identity/v3/role_assignments?user.id=65508fe717ca4eb996135023e2296dd1&scope.project.id=e6d05dd0b455478c8762a69b412f4a4a&role.id=34235948f8f943a99328516c2171cfff => generated 259 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:43.480136 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a27d7de-30c8-4964-a120-d6835734ded2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:43.480981 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a27d7de-30c8-4964-a120-d6835734ded2 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:43.481362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a27d7de-30c8-4964-a120-d6835734ded2 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:43.481747 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a27d7de-30c8-4964-a120-d6835734ded2 None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:43.483182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a27d7de-30c8-4964-a120-d6835734ded2 None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:43.484892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a27d7de-30c8-4964-a120-d6835734ded2 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:43.487092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 48/96] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:43 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:43.493015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9152687b-4482-4401-8dcc-9f8e3acee7a0 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:43.544552 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:43.648178 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9152687b-4482-4401-8dcc-9f8e3acee7a0 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:43.699517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87eb887e-f3c8-45c5-9514-a0e822a51c07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:43.700527 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87eb887e-f3c8-45c5-9514-a0e822a51c07 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:43.701015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87eb887e-f3c8-45c5-9514-a0e822a51c07 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:43.701493 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87eb887e-f3c8-45c5-9514-a0e822a51c07 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:43.702479 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87eb887e-f3c8-45c5-9514-a0e822a51c07 None admin] RBAC: Adding query filter params (role.id=285107fb9b2449fe9f1d29f3075a246f, scope.project.id=d6715600d626423b8e8a06d45bb53d08, user.id=65508fe717ca4eb996135023e2296dd1) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:43.703024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87eb887e-f3c8-45c5-9514-a0e822a51c07 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:43.705365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87eb887e-f3c8-45c5-9514-a0e822a51c07 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:43.711994 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 49/97] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:46:43 2022] GET /identity/v3/role_assignments?user.id=65508fe717ca4eb996135023e2296dd1&scope.project.id=d6715600d626423b8e8a06d45bb53d08&role.id=285107fb9b2449fe9f1d29f3075a246f => generated 259 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:45.201583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0dcc8d-039f-4b65-a664-3275d53301a1 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:45.202186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0dcc8d-039f-4b65-a664-3275d53301a1 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:45.202186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0dcc8d-039f-4b65-a664-3275d53301a1 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:45.204384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 49/98] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:45.209790 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ece79b-ae16-4327-8da1-2f758fdd46c8 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:45.210674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ece79b-ae16-4327-8da1-2f758fdd46c8 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:45.211172 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ece79b-ae16-4327-8da1-2f758fdd46c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:45.215394 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86fab37-626e-4a31-8048-24831c828eab None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:45.216317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86fab37-626e-4a31-8048-24831c828eab None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:45.217319 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86fab37-626e-4a31-8048-24831c828eab None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:45.220494 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 50/99] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:45 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:45.229996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af296476-6ae5-420e-bab4-7fce2826f462 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:45.230568 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af296476-6ae5-420e-bab4-7fce2826f462 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:45.231043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af296476-6ae5-420e-bab4-7fce2826f462 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:45.261700 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-b7ece79b-ae16-4327-8da1-2f758fdd46c8 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:45.263161 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-b7ece79b-ae16-4327-8da1-2f758fdd46c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:45.273239 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-af296476-6ae5-420e-bab4-7fce2826f462 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:45.278774 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-af296476-6ae5-420e-bab4-7fce2826f462 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:45.404207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 50/100] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:45.410213 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 51/101] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:46:45.795286 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ece79b-ae16-4327-8da1-2f758fdd46c8 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:45.830307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:45.929796 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ece79b-ae16-4327-8da1-2f758fdd46c8 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:45.939533 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af296476-6ae5-420e-bab4-7fce2826f462 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:45.998826 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa104625-84ea-46e6-b818-6de2d5b0aa94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:45.999480 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa104625-84ea-46e6-b818-6de2d5b0aa94 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:45.999767 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa104625-84ea-46e6-b818-6de2d5b0aa94 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:46.000087 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa104625-84ea-46e6-b818-6de2d5b0aa94 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:46.006866 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa104625-84ea-46e6-b818-6de2d5b0aa94 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:46.008358 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa104625-84ea-46e6-b818-6de2d5b0aa94 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:46.010008 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 51/102] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:46:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:46.024724 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:46.029621 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa104625-84ea-46e6-b818-6de2d5b0aa94 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:46.075701 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af296476-6ae5-420e-bab4-7fce2826f462 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:46.145503 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ad5b20-bfc6-4088-9cb6-a4f1bbfedaaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:46.146165 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ad5b20-bfc6-4088-9cb6-a4f1bbfedaaf None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:46.146456 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ad5b20-bfc6-4088-9cb6-a4f1bbfedaaf None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:46.146730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ad5b20-bfc6-4088-9cb6-a4f1bbfedaaf None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:46.178752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ad5b20-bfc6-4088-9cb6-a4f1bbfedaaf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:46.183714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-68ad5b20-bfc6-4088-9cb6-a4f1bbfedaaf None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:46:46.183714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:46.183714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:46.183714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:46.183714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:46.183714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:46.183714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:46.183714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:46.183714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:46.183714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:46.183714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:46.183714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:46.183714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:46.183714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:46.183714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:46.183714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:46.184551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:46.184551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:46.184551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:46.184551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:46.184551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:46.184551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:46.184551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:46.184551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:46.184551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:46.184551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:46.184551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:46.184551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:46.184551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:46.184551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:46.184551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:46.184551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:46.185288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:46.185288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:46.185288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:46.185288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:46.185288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:46.185288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:46.185288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:46.185288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:46.185288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:46.185288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:46.185288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:46:46.185288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:46:46.185288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 52/103] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:46:45 2022] GET /identity/v3/roles/reader => generated 84 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:46:46.191789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ad5b20-bfc6-4088-9cb6-a4f1bbfedaaf None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:46.217057 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:46.241488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa104625-84ea-46e6-b818-6de2d5b0aa94 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:46.320952 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e004e655-f83e-43b9-8db3-cae3df2f5e02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:46.321589 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e004e655-f83e-43b9-8db3-cae3df2f5e02 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:46.321871 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e004e655-f83e-43b9-8db3-cae3df2f5e02 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:46.322173 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e004e655-f83e-43b9-8db3-cae3df2f5e02 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:46.323227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e004e655-f83e-43b9-8db3-cae3df2f5e02 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:46.324947 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e004e655-f83e-43b9-8db3-cae3df2f5e02 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:46.392045 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 52/104] 10.222.0.27 () {60 vars in 1141 bytes} [Wed Sep 7 03:46:46 2022] POST /identity/v3/projects => generated 302 bytes in 363 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:46:46.455108 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ad5b20-bfc6-4088-9cb6-a4f1bbfedaaf None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:46.491282 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:46.496184 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad8e537-be4b-4670-8796-dc2f9c6a955d None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:46.496837 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad8e537-be4b-4670-8796-dc2f9c6a955d None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:46.497476 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad8e537-be4b-4670-8796-dc2f9c6a955d None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:46.499747 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 53/105] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:46.509157 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6142bd85-e010-49be-aa45-fda5167f3ce0 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:46.509746 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6142bd85-e010-49be-aa45-fda5167f3ce0 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:46.510246 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6142bd85-e010-49be-aa45-fda5167f3ce0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:46.529995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3087a5-591d-4d61-aa05-a7e4f35709af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:46.531040 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3087a5-591d-4d61-aa05-a7e4f35709af None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:46.531755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3087a5-591d-4d61-aa05-a7e4f35709af None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:46.532237 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3087a5-591d-4d61-aa05-a7e4f35709af None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:46.533143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3087a5-591d-4d61-aa05-a7e4f35709af None admin] RBAC: Adding query filter params (name=reader) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:46.533681 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3087a5-591d-4d61-aa05-a7e4f35709af None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:46.535686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3087a5-591d-4d61-aa05-a7e4f35709af None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:46.550561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-6142bd85-e010-49be-aa45-fda5167f3ce0 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:46.552340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-6142bd85-e010-49be-aa45-fda5167f3ce0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:46.558599 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 53/106] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:46:46 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 367 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:46.565001 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2deebbcc-e3a0-46bf-bc70-e26931ed16aa None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:46.565596 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2deebbcc-e3a0-46bf-bc70-e26931ed16aa None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:46.566081 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2deebbcc-e3a0-46bf-bc70-e26931ed16aa None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:46.571362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 54/107] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:46 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:46.572527 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2deebbcc-e3a0-46bf-bc70-e26931ed16aa None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:46.694991 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 54/108] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:46.696620 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf70d89a-a572-4112-b2d4-9f9f9381771f None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:46.697111 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf70d89a-a572-4112-b2d4-9f9f9381771f None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:46.697591 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf70d89a-a572-4112-b2d4-9f9f9381771f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:46.719944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-bf70d89a-a572-4112-b2d4-9f9f9381771f None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:46.721712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-bf70d89a-a572-4112-b2d4-9f9f9381771f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:46.749295 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2deebbcc-e3a0-46bf-bc70-e26931ed16aa None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:46.793400 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b3341a-4ada-4cc2-80bd-63bca9766e2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:46.794307 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b3341a-4ada-4cc2-80bd-63bca9766e2f None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:46.794588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b3341a-4ada-4cc2-80bd-63bca9766e2f None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:46.794866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b3341a-4ada-4cc2-80bd-63bca9766e2f None admin] PATH_INFO: `/v3/users/b4095e120d9b4f6a8a1c912193b8823f` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:46.809773 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 55/109] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:46.811639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb37b8d-d606-46ac-b346-52ada9849094 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:46.812130 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb37b8d-d606-46ac-b346-52ada9849094 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:46.812774 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb37b8d-d606-46ac-b346-52ada9849094 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:46.815280 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 56/110] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:46 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:46.818824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b3341a-4ada-4cc2-80bd-63bca9766e2f None admin] RBAC: Authorizing `identity:get_user(user_id=b4095e120d9b4f6a8a1c912193b8823f)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:46.820398 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a74b6e-81de-4317-9279-f962b410e2ff None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:46.820522 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b3341a-4ada-4cc2-80bd-63bca9766e2f None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:46.820999 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a74b6e-81de-4317-9279-f962b410e2ff None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:46.821519 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a74b6e-81de-4317-9279-f962b410e2ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:46.822403 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 55/111] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:46 2022] GET /identity/v3/users/b4095e120d9b4f6a8a1c912193b8823f => generated 289 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:46.828626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b3341a-4ada-4cc2-80bd-63bca9766e2f None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:46.841322 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:46.844666 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-35a74b6e-81de-4317-9279-f962b410e2ff None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:46.846535 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-35a74b6e-81de-4317-9279-f962b410e2ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:46.881666 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:46.927133 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 57/112] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:46:46.943363 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b3341a-4ada-4cc2-80bd-63bca9766e2f None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:46.981109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d37e7c5-c938-4288-8d99-40147b1189ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:46.981945 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d37e7c5-c938-4288-8d99-40147b1189ed None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:46.982392 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d37e7c5-c938-4288-8d99-40147b1189ed None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:46.982829 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d37e7c5-c938-4288-8d99-40147b1189ed None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:47.053271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d37e7c5-c938-4288-8d99-40147b1189ed None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:47.055072 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d37e7c5-c938-4288-8d99-40147b1189ed None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:47.105892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:47.163053 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 56/113] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:46:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:46:47.169126 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a74b6e-81de-4317-9279-f962b410e2ff None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:47.239756 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d37e7c5-c938-4288-8d99-40147b1189ed None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:47.296695 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a74b6e-81de-4317-9279-f962b410e2ff None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:47.335262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5840afe-b781-4742-99cd-b6bef47f294a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:47.336241 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5840afe-b781-4742-99cd-b6bef47f294a None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:47.336690 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5840afe-b781-4742-99cd-b6bef47f294a None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:47.337107 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5840afe-b781-4742-99cd-b6bef47f294a None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:47.338410 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5840afe-b781-4742-99cd-b6bef47f294a None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:47.340107 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5840afe-b781-4742-99cd-b6bef47f294a None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:47.342256 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 58/114] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:47 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:47.343347 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5840afe-b781-4742-99cd-b6bef47f294a None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:47.352277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:47.407124 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d37e7c5-c938-4288-8d99-40147b1189ed None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:47.444567 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20dd3a69-d92e-40b8-89e0-314324dde467 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:47.445207 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dd3a69-d92e-40b8-89e0-314324dde467 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:47.445639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dd3a69-d92e-40b8-89e0-314324dde467 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:47.445918 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dd3a69-d92e-40b8-89e0-314324dde467 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:47.456996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20dd3a69-d92e-40b8-89e0-314324dde467 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:47.457551 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5840afe-b781-4742-99cd-b6bef47f294a None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:47.461838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-20dd3a69-d92e-40b8-89e0-314324dde467 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:46:47.461838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:47.461838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:47.461838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:47.461838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:47.461838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:47.461838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:47.461838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:47.461838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:47.461838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:47.461838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:47.461838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:47.461838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:47.461838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:47.461838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:47.461838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:47.462883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:47.462883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:47.462883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:47.462883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:47.462883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:47.462883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:47.462883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:47.462883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:47.462883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:47.462883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:47.462883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:47.462883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:47.462883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:47.462883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:47.462883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:47.462883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:47.468939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:47.468939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:47.468939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:47.468939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:47.468939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:47.468939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:47.468939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:47.468939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:47.468939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:47.468939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:47.468939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:46:47.468939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:46:47.468939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 57/115] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:46:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:47.468939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20dd3a69-d92e-40b8-89e0-314324dde467 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:47.476408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:47.501660 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7b8bc4-589f-4cd9-a878-2e4c12b93bcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:47.502261 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7b8bc4-589f-4cd9-a878-2e4c12b93bcf None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:47.502668 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7b8bc4-589f-4cd9-a878-2e4c12b93bcf None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:47.502950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7b8bc4-589f-4cd9-a878-2e4c12b93bcf None admin] PATH_INFO: `/v3/roles/member` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:47.513662 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7b8bc4-589f-4cd9-a878-2e4c12b93bcf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:47.519317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-ec7b8bc4-589f-4cd9-a878-2e4c12b93bcf None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:46:47.519317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:47.519317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:47.519317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:47.519317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:47.519317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:47.519317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:47.519317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:47.519317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:47.519317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:47.519317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:47.519317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:47.519317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:47.519317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:47.519317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:47.519317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:47.520152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:47.520152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:47.520152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:47.520152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:47.520152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:47.520152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:47.520152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:47.520152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:47.520152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:47.520152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:47.520152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:47.520152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:47.520152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:47.520152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:47.520152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:47.520152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:47.520897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:47.520897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:47.520897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:47.520897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:47.520897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:47.520897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:47.520897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:47.520897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:47.520897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:47.520897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:47.520897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:46:47.520897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:46:47.520897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 59/116] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:46:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:47.522047 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7b8bc4-589f-4cd9-a878-2e4c12b93bcf None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:47.577864 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20dd3a69-d92e-40b8-89e0-314324dde467 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:47.588520 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:47.615027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda9bee4-4ebd-47a5-a5cb-1ec407dcbb3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:47.615629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda9bee4-4ebd-47a5-a5cb-1ec407dcbb3a None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:47.615908 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda9bee4-4ebd-47a5-a5cb-1ec407dcbb3a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:47.616181 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda9bee4-4ebd-47a5-a5cb-1ec407dcbb3a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:47.616962 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda9bee4-4ebd-47a5-a5cb-1ec407dcbb3a None admin] RBAC: Adding query filter params (role.id=e8be279b370349c8a5b9a06dd763e1b5, scope.project.id=d6715600d626423b8e8a06d45bb53d08, user.id=b4095e120d9b4f6a8a1c912193b8823f) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:47.617212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda9bee4-4ebd-47a5-a5cb-1ec407dcbb3a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:47.618682 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda9bee4-4ebd-47a5-a5cb-1ec407dcbb3a None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:47.624003 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 58/117] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:46:47 2022] GET /identity/v3/role_assignments?user.id=b4095e120d9b4f6a8a1c912193b8823f&scope.project.id=d6715600d626423b8e8a06d45bb53d08&role.id=e8be279b370349c8a5b9a06dd763e1b5 => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:47.625306 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e58f344-5d63-4c6c-805a-070042843d94 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:47.625595 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e58f344-5d63-4c6c-805a-070042843d94 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:47.625867 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e58f344-5d63-4c6c-805a-070042843d94 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:47.627266 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 59/118] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:47.627846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e58f344-5d63-4c6c-805a-070042843d94 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:47.642394 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7b8bc4-589f-4cd9-a878-2e4c12b93bcf None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:47.681938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79227a46-d7bd-41de-8599-9d6dbcc32e7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:47.682655 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79227a46-d7bd-41de-8599-9d6dbcc32e7e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:47.682935 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79227a46-d7bd-41de-8599-9d6dbcc32e7e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:47.683341 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79227a46-d7bd-41de-8599-9d6dbcc32e7e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:47.698692 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79227a46-d7bd-41de-8599-9d6dbcc32e7e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:47.706156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-79227a46-d7bd-41de-8599-9d6dbcc32e7e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:46:47.706156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:47.706156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:47.706156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:47.706156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:47.706156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:47.706156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:47.706156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:47.706156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:47.706156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:47.706156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:47.706156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:47.706156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:47.706156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:47.706156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:47.706156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:47.707189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:47.707189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:47.707189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:47.707189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:47.707189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:47.707189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:47.707189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:47.707189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:47.707189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:47.707189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:47.707189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:47.707189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:47.707189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:47.707189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:47.707189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:47.707189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:47.707951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:47.707951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:47.707951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:47.707951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:47.707951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:47.707951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:47.707951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:47.707951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:47.707951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:47.707951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:47.707951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:46:47.707951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:46:47.707951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 60/119] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:46:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:47.708713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79227a46-d7bd-41de-8599-9d6dbcc32e7e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:47.739256 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:47.782812 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e58f344-5d63-4c6c-805a-070042843d94 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:47.821773 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a13b54-e434-4a82-9589-a3f4b1984cb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:47.822499 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a13b54-e434-4a82-9589-a3f4b1984cb9 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:47.822787 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a13b54-e434-4a82-9589-a3f4b1984cb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:47.823067 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a13b54-e434-4a82-9589-a3f4b1984cb9 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:47.823863 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a13b54-e434-4a82-9589-a3f4b1984cb9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:47.824289 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a13b54-e434-4a82-9589-a3f4b1984cb9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:47.825660 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a13b54-e434-4a82-9589-a3f4b1984cb9 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:47.833584 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79227a46-d7bd-41de-8599-9d6dbcc32e7e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:47.837129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 60/120] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:46:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:47.837914 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c474359-13d3-459d-a80e-eec3abf1d552 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:47.838240 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c474359-13d3-459d-a80e-eec3abf1d552 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:47.838525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c474359-13d3-459d-a80e-eec3abf1d552 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:47.857304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-7c474359-13d3-459d-a80e-eec3abf1d552 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:47.858549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-7c474359-13d3-459d-a80e-eec3abf1d552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:47.874479 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:47.878731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59863232-1c47-48c5-9516-65670acdd9c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:47.879417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59863232-1c47-48c5-9516-65670acdd9c4 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:47.879707 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59863232-1c47-48c5-9516-65670acdd9c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:47.880047 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59863232-1c47-48c5-9516-65670acdd9c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:47.880949 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59863232-1c47-48c5-9516-65670acdd9c4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:47.881263 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59863232-1c47-48c5-9516-65670acdd9c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:47.883204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59863232-1c47-48c5-9516-65670acdd9c4 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:47.894623 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 61/121] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:46:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:47.899437 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59863232-1c47-48c5-9516-65670acdd9c4 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:47.958452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 61/122] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:47.958452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c474359-13d3-459d-a80e-eec3abf1d552 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:48.065864 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59863232-1c47-48c5-9516-65670acdd9c4 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:48.102634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614b8f42-9e90-4abd-92d7-d4bf5e7869b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:48.103776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614b8f42-9e90-4abd-92d7-d4bf5e7869b5 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:48.104252 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614b8f42-9e90-4abd-92d7-d4bf5e7869b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:48.104739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:48.104953 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614b8f42-9e90-4abd-92d7-d4bf5e7869b5 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:48.106264 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614b8f42-9e90-4abd-92d7-d4bf5e7869b5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:48.106798 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614b8f42-9e90-4abd-92d7-d4bf5e7869b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:48.108651 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614b8f42-9e90-4abd-92d7-d4bf5e7869b5 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:48.118318 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 62/123] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:46:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:48.119485 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614b8f42-9e90-4abd-92d7-d4bf5e7869b5 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:48.183990 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:48.186763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c474359-13d3-459d-a80e-eec3abf1d552 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:48.247929 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f170dc-497a-4c15-af14-9c87cbb75051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:48.248602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f170dc-497a-4c15-af14-9c87cbb75051 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:48.248881 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f170dc-497a-4c15-af14-9c87cbb75051 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:48.249153 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f170dc-497a-4c15-af14-9c87cbb75051 None admin] PATH_INFO: `/v3/users/c81176b6299a493294b869b497bad7f4` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:48.250377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f170dc-497a-4c15-af14-9c87cbb75051 None admin] RBAC: Authorizing `identity:get_user(user_id=c81176b6299a493294b869b497bad7f4)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:48.251899 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f170dc-497a-4c15-af14-9c87cbb75051 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:48.253488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 62/124] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:47 2022] GET /identity/v3/users/c81176b6299a493294b869b497bad7f4 => generated 247 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:48.254294 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f170dc-497a-4c15-af14-9c87cbb75051 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:48.270889 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614b8f42-9e90-4abd-92d7-d4bf5e7869b5 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:48.309705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa96d79-c4c4-47d5-a113-9628df871a61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:48.310470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa96d79-c4c4-47d5-a113-9628df871a61 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:48.310698 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa96d79-c4c4-47d5-a113-9628df871a61 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:48.310977 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa96d79-c4c4-47d5-a113-9628df871a61 None admin] PATH_INFO: `/v3/users/65508fe717ca4eb996135023e2296dd1` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:48.312195 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa96d79-c4c4-47d5-a113-9628df871a61 None admin] RBAC: Authorizing `identity:get_user(user_id=65508fe717ca4eb996135023e2296dd1)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:48.314786 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa96d79-c4c4-47d5-a113-9628df871a61 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:48.320725 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 63/125] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:48 2022] GET /identity/v3/users/65508fe717ca4eb996135023e2296dd1 => generated 275 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:48.320725 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa96d79-c4c4-47d5-a113-9628df871a61 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:48.375983 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f170dc-497a-4c15-af14-9c87cbb75051 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:48.407484 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:48.416799 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b2a1a9-67dd-4a3d-a5f9-dac729d90883 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:48.417775 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b2a1a9-67dd-4a3d-a5f9-dac729d90883 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:48.418198 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b2a1a9-67dd-4a3d-a5f9-dac729d90883 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:48.418579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b2a1a9-67dd-4a3d-a5f9-dac729d90883 None admin] PATH_INFO: `/v3/users/65508fe717ca4eb996135023e2296dd1` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:48.419871 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b2a1a9-67dd-4a3d-a5f9-dac729d90883 None admin] RBAC: Authorizing `identity:get_user(user_id=65508fe717ca4eb996135023e2296dd1)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:48.421461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b2a1a9-67dd-4a3d-a5f9-dac729d90883 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:48.423792 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 63/126] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:48 2022] GET /identity/v3/users/65508fe717ca4eb996135023e2296dd1 => generated 275 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:48.424650 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b2a1a9-67dd-4a3d-a5f9-dac729d90883 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:48.443325 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa96d79-c4c4-47d5-a113-9628df871a61 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:48.462739 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:48.486844 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0338ac9-6e06-43a8-8214-47031af557c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:48.487546 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0338ac9-6e06-43a8-8214-47031af557c3 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:48.487836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0338ac9-6e06-43a8-8214-47031af557c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:48.488119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0338ac9-6e06-43a8-8214-47031af557c3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:48.501431 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0338ac9-6e06-43a8-8214-47031af557c3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:48.507012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-b0338ac9-6e06-43a8-8214-47031af557c3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:46:48.507012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:48.507012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:48.507012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:48.507012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:48.507012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:48.507012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:48.507012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:48.507012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:48.507012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:48.507012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:48.507012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:48.507012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:48.507012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:48.507012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:48.507012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:48.507846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:48.507846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:48.507846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:48.507846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:48.507846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:48.507846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:48.507846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:48.507846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:48.507846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:48.507846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:48.507846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:48.507846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:48.507846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:48.507846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:48.507846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:48.507846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:48.508598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:48.508598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:48.508598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:48.508598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:48.508598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:48.508598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:48.508598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:48.508598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:48.508598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:48.508598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:48.508598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:46:48.508598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:46:48.508598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 64/127] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:46:48 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:48.510191 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0338ac9-6e06-43a8-8214-47031af557c3 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:48.570014 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b2a1a9-67dd-4a3d-a5f9-dac729d90883 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:48.611602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c10430f-caec-4084-be13-b6c74b252fbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:48.612255 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c10430f-caec-4084-be13-b6c74b252fbc None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:48.612541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c10430f-caec-4084-be13-b6c74b252fbc None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:48.612821 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c10430f-caec-4084-be13-b6c74b252fbc None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:48.614038 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c10430f-caec-4084-be13-b6c74b252fbc None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:48.615603 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c10430f-caec-4084-be13-b6c74b252fbc None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:48.617672 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 64/128] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:48 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:48.618555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c10430f-caec-4084-be13-b6c74b252fbc None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:48.626574 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0338ac9-6e06-43a8-8214-47031af557c3 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:48.663496 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30105972-8af3-4040-a57f-109d68a3302b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:48.664377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30105972-8af3-4040-a57f-109d68a3302b None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:48.664665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30105972-8af3-4040-a57f-109d68a3302b None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:48.664944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30105972-8af3-4040-a57f-109d68a3302b None admin] PATH_INFO: `/v3/projects/e6d05dd0b455478c8762a69b412f4a4a` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:48.666109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30105972-8af3-4040-a57f-109d68a3302b None admin] RBAC: Authorizing `identity:get_project(project_id=e6d05dd0b455478c8762a69b412f4a4a)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:48.667631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30105972-8af3-4040-a57f-109d68a3302b None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:48.669443 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 65/129] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:48 2022] GET /identity/v3/projects/e6d05dd0b455478c8762a69b412f4a4a => generated 312 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:48.670365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30105972-8af3-4040-a57f-109d68a3302b None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:48.714538 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:48.721384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:48.764078 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c10430f-caec-4084-be13-b6c74b252fbc None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:48.790556 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30105972-8af3-4040-a57f-109d68a3302b None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:48.815218 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a89cea-984d-4db2-a145-e41fa2fdc245 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:48.816200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a89cea-984d-4db2-a145-e41fa2fdc245 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:48.816487 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a89cea-984d-4db2-a145-e41fa2fdc245 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:48.816764 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a89cea-984d-4db2-a145-e41fa2fdc245 None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:48.818591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a89cea-984d-4db2-a145-e41fa2fdc245 None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:48.820652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a89cea-984d-4db2-a145-e41fa2fdc245 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:48.823506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 65/130] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:48 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:48.824344 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a89cea-984d-4db2-a145-e41fa2fdc245 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:48.831130 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81490dfc-58a6-410d-a402-ac066a3e952e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:48.831767 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81490dfc-58a6-410d-a402-ac066a3e952e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:48.832052 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81490dfc-58a6-410d-a402-ac066a3e952e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:48.832332 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81490dfc-58a6-410d-a402-ac066a3e952e None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:48.833021 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81490dfc-58a6-410d-a402-ac066a3e952e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:48.833333 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81490dfc-58a6-410d-a402-ac066a3e952e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:48.834782 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81490dfc-58a6-410d-a402-ac066a3e952e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:48.849050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 66/131] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:46:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:48.849990 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81490dfc-58a6-410d-a402-ac066a3e952e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:48.933662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a89cea-984d-4db2-a145-e41fa2fdc245 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:48.958125 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81490dfc-58a6-410d-a402-ac066a3e952e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:48.960569 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:48.970500 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1683bbba-018c-4aea-8d86-893ae8f73036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:48.971173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1683bbba-018c-4aea-8d86-893ae8f73036 None admin] REQUEST_METHOD: `PUT` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:48.971555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1683bbba-018c-4aea-8d86-893ae8f73036 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:48.971837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1683bbba-018c-4aea-8d86-893ae8f73036 None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08/users/c81176b6299a493294b869b497bad7f4/roles/9d5b19a61c46475d8048b1bd66ca25a1` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:48.973884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1683bbba-018c-4aea-8d86-893ae8f73036 None admin] RBAC: Authorizing `identity:create_grant(project_id=d6715600d626423b8e8a06d45bb53d08, user_id=c81176b6299a493294b869b497bad7f4, role_id=9d5b19a61c46475d8048b1bd66ca25a1)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:48.975635 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1683bbba-018c-4aea-8d86-893ae8f73036 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:48.978071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:48.996520 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 66/132] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:46:48 2022] PUT /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08/users/c81176b6299a493294b869b497bad7f4/roles/9d5b19a61c46475d8048b1bd66ca25a1 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:46:48.997466 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1683bbba-018c-4aea-8d86-893ae8f73036 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:49.003788 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702d7160-4989-47ee-9919-8e939a03b262 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:49.004509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702d7160-4989-47ee-9919-8e939a03b262 None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:49.004800 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702d7160-4989-47ee-9919-8e939a03b262 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:49.005090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702d7160-4989-47ee-9919-8e939a03b262 None admin] PATH_INFO: `/v3/projects/e6d05dd0b455478c8762a69b412f4a4a/users/65508fe717ca4eb996135023e2296dd1/roles/34235948f8f943a99328516c2171cfff` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:49.011869 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702d7160-4989-47ee-9919-8e939a03b262 None admin] RBAC: Authorizing `identity:create_grant(project_id=e6d05dd0b455478c8762a69b412f4a4a, user_id=65508fe717ca4eb996135023e2296dd1, role_id=34235948f8f943a99328516c2171cfff)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:49.013530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702d7160-4989-47ee-9919-8e939a03b262 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:49.049339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 67/133] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:46:48 2022] PUT /identity/v3/projects/e6d05dd0b455478c8762a69b412f4a4a/users/65508fe717ca4eb996135023e2296dd1/roles/34235948f8f943a99328516c2171cfff => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:46:49.050314 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702d7160-4989-47ee-9919-8e939a03b262 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:49.157294 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1683bbba-018c-4aea-8d86-893ae8f73036 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:49.198446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e80af6-cdcb-4b49-b58f-7154fb1281f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:49.199055 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e80af6-cdcb-4b49-b58f-7154fb1281f5 None admin] REQUEST_METHOD: `PUT` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:49.199477 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e80af6-cdcb-4b49-b58f-7154fb1281f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:49.199764 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e80af6-cdcb-4b49-b58f-7154fb1281f5 None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08/users/65508fe717ca4eb996135023e2296dd1/roles/34235948f8f943a99328516c2171cfff` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:49.201873 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e80af6-cdcb-4b49-b58f-7154fb1281f5 None admin] RBAC: Authorizing `identity:create_grant(project_id=d6715600d626423b8e8a06d45bb53d08, user_id=65508fe717ca4eb996135023e2296dd1, role_id=34235948f8f943a99328516c2171cfff)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:49.204347 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e80af6-cdcb-4b49-b58f-7154fb1281f5 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:49.227384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 67/134] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:46:48 2022] PUT /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08/users/65508fe717ca4eb996135023e2296dd1/roles/34235948f8f943a99328516c2171cfff => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:46:49.230017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7a1580-cd54-4542-8fe8-925ceadf453c None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:49.230331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7a1580-cd54-4542-8fe8-925ceadf453c None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:49.231441 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7a1580-cd54-4542-8fe8-925ceadf453c None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:49.235492 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 68/135] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:49 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:49.247100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20b2ed3-a42f-4d74-a21d-914e4e2a9746 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:49.247411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20b2ed3-a42f-4d74-a21d-914e4e2a9746 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:49.247659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20b2ed3-a42f-4d74-a21d-914e4e2a9746 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:49.292321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-b20b2ed3-a42f-4d74-a21d-914e4e2a9746 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:49.296597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-b20b2ed3-a42f-4d74-a21d-914e4e2a9746 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:49.341667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702d7160-4989-47ee-9919-8e939a03b262 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:49.380436 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:49.400192 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd5d01a-9be4-451d-af11-a89377c71a41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:49.401316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd5d01a-9be4-451d-af11-a89377c71a41 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:49.401864 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd5d01a-9be4-451d-af11-a89377c71a41 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:49.402179 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd5d01a-9be4-451d-af11-a89377c71a41 None admin] PATH_INFO: `/v3/users/65508fe717ca4eb996135023e2296dd1` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:49.403447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd5d01a-9be4-451d-af11-a89377c71a41 None admin] RBAC: Authorizing `identity:get_user(user_id=65508fe717ca4eb996135023e2296dd1)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:49.404945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd5d01a-9be4-451d-af11-a89377c71a41 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:49.406944 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 69/136] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:46:49.407381 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 68/137] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:49 2022] GET /identity/v3/users/65508fe717ca4eb996135023e2296dd1 => generated 275 bytes in 357 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:49.413222 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20b2ed3-a42f-4d74-a21d-914e4e2a9746 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:49.524914 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20b2ed3-a42f-4d74-a21d-914e4e2a9746 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:49.563292 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce26823-8d5c-4b08-b643-3c290640a50f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:49.563937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce26823-8d5c-4b08-b643-3c290640a50f None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:49.564221 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce26823-8d5c-4b08-b643-3c290640a50f None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:49.564497 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce26823-8d5c-4b08-b643-3c290640a50f None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:49.565670 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce26823-8d5c-4b08-b643-3c290640a50f None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:49.567224 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce26823-8d5c-4b08-b643-3c290640a50f None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:49.568887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 70/138] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:49 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:49.574619 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd5d01a-9be4-451d-af11-a89377c71a41 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:49.609968 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:49.683917 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd5d01a-9be4-451d-af11-a89377c71a41 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:49.721023 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b828b0-feb1-4755-a608-90a05c49f172 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:49.721703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b828b0-feb1-4755-a608-90a05c49f172 None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:49.721987 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b828b0-feb1-4755-a608-90a05c49f172 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:49.722290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b828b0-feb1-4755-a608-90a05c49f172 None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08/users/65508fe717ca4eb996135023e2296dd1/roles/285107fb9b2449fe9f1d29f3075a246f` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:49.724583 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b828b0-feb1-4755-a608-90a05c49f172 None admin] RBAC: Authorizing `identity:create_grant(project_id=d6715600d626423b8e8a06d45bb53d08, user_id=65508fe717ca4eb996135023e2296dd1, role_id=285107fb9b2449fe9f1d29f3075a246f)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:49.726055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b828b0-feb1-4755-a608-90a05c49f172 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:49.744754 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 69/139] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:46:49 2022] PUT /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08/users/65508fe717ca4eb996135023e2296dd1/roles/285107fb9b2449fe9f1d29f3075a246f => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:46:49.846162 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce26823-8d5c-4b08-b643-3c290640a50f None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:49.919778 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:49.972589 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce26823-8d5c-4b08-b643-3c290640a50f None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:50.011698 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db59e0ce-1e3d-4a8c-a0bd-3569df4f36d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:50.012362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db59e0ce-1e3d-4a8c-a0bd-3569df4f36d3 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:50.012643 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db59e0ce-1e3d-4a8c-a0bd-3569df4f36d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:50.012919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db59e0ce-1e3d-4a8c-a0bd-3569df4f36d3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:50.014138 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db59e0ce-1e3d-4a8c-a0bd-3569df4f36d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:50.016235 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db59e0ce-1e3d-4a8c-a0bd-3569df4f36d3 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:50.018167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 71/140] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:46:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:50.024064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b828b0-feb1-4755-a608-90a05c49f172 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:50.130402 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:50.139937 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b828b0-feb1-4755-a608-90a05c49f172 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:50.179253 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654e994e-8a64-4fab-ab6a-8468af7aa92b None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:50.179544 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654e994e-8a64-4fab-ab6a-8468af7aa92b None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:50.179794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654e994e-8a64-4fab-ab6a-8468af7aa92b None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:50.181261 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 72/141] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:46:50.181544 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4737fd-e04b-455d-8c1b-59d73436e38a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:50.182233 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4737fd-e04b-455d-8c1b-59d73436e38a None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:50.182569 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4737fd-e04b-455d-8c1b-59d73436e38a None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:50.182847 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4737fd-e04b-455d-8c1b-59d73436e38a None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:50.183530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4737fd-e04b-455d-8c1b-59d73436e38a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:50.185005 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4737fd-e04b-455d-8c1b-59d73436e38a None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:50.186368 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e79a950-8ec3-4491-8fe4-1245bf134357 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:50.186649 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e79a950-8ec3-4491-8fe4-1245bf134357 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:50.186924 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e79a950-8ec3-4491-8fe4-1245bf134357 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:50.220647 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-0e79a950-8ec3-4491-8fe4-1245bf134357 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:50.221999 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-0e79a950-8ec3-4491-8fe4-1245bf134357 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:50.251209 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 70/142] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:46:50 2022] POST /identity/v3/users => generated 283 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:46:50.342438 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 73/143] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:46:50.881476 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4737fd-e04b-455d-8c1b-59d73436e38a None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:51.001762 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4737fd-e04b-455d-8c1b-59d73436e38a None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:51.049942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1a990e-ab4d-4d06-984a-da0cb423780c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:51.050658 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1a990e-ab4d-4d06-984a-da0cb423780c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:51.050942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1a990e-ab4d-4d06-984a-da0cb423780c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:51.051284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1a990e-ab4d-4d06-984a-da0cb423780c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:51.065580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1a990e-ab4d-4d06-984a-da0cb423780c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:51.070604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-5a1a990e-ab4d-4d06-984a-da0cb423780c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:46:51.070604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:51.070604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:51.070604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:51.070604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:51.070604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:51.070604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:51.070604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:51.070604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:51.070604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:51.070604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:51.070604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:51.070604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:51.070604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:51.070604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:51.070604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:51.071409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:51.071409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:51.071409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:51.071409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:51.071409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:51.071409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:51.071409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:51.071409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:51.071409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:51.071409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:51.071409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:51.071409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:51.071409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:51.071409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:51.071409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:51.071409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:51.072150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:51.072150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:51.072150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:51.072150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:51.072150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:51.072150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:51.072150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:51.072150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:51.072150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:51.072150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:51.072150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:46:51.072150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:46:51.072150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 71/144] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:46:50 2022] GET /identity/v3/roles/reader => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:51.077511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e79a950-8ec3-4491-8fe4-1245bf134357 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:51.157575 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:51.212314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e79a950-8ec3-4491-8fe4-1245bf134357 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:51.257939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462f4dac-434f-4dc2-ade2-e8df1f988dc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:51.258595 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462f4dac-434f-4dc2-ade2-e8df1f988dc8 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:51.258872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462f4dac-434f-4dc2-ade2-e8df1f988dc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:51.259141 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462f4dac-434f-4dc2-ade2-e8df1f988dc8 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:51.259812 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462f4dac-434f-4dc2-ade2-e8df1f988dc8 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:51.260391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462f4dac-434f-4dc2-ade2-e8df1f988dc8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:51.266136 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462f4dac-434f-4dc2-ade2-e8df1f988dc8 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:51.271312 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 74/145] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:46:51 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:51.276756 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1a990e-ab4d-4d06-984a-da0cb423780c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:51.300885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:51.410487 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1a990e-ab4d-4d06-984a-da0cb423780c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:51.449129 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ab3347-654c-4b13-a379-985ae47c19bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:51.449770 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ab3347-654c-4b13-a379-985ae47c19bc None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:51.450050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ab3347-654c-4b13-a379-985ae47c19bc None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:51.450353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ab3347-654c-4b13-a379-985ae47c19bc None admin] PATH_INFO: `/v3/users/b4095e120d9b4f6a8a1c912193b8823f` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:51.451504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ab3347-654c-4b13-a379-985ae47c19bc None admin] RBAC: Authorizing `identity:get_user(user_id=b4095e120d9b4f6a8a1c912193b8823f)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:51.453742 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ab3347-654c-4b13-a379-985ae47c19bc None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:51.455429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 72/146] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:51 2022] GET /identity/v3/users/b4095e120d9b4f6a8a1c912193b8823f => generated 289 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:51.461613 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462f4dac-434f-4dc2-ade2-e8df1f988dc8 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:51.585956 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462f4dac-434f-4dc2-ade2-e8df1f988dc8 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:51.599457 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:51.624242 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16bf879-e542-441d-9b97-624bd2e38ef3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:51.625401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16bf879-e542-441d-9b97-624bd2e38ef3 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:51.625756 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16bf879-e542-441d-9b97-624bd2e38ef3 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:51.626059 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16bf879-e542-441d-9b97-624bd2e38ef3 None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:51.627220 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16bf879-e542-441d-9b97-624bd2e38ef3 None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:51.628660 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16bf879-e542-441d-9b97-624bd2e38ef3 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:51.630622 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 75/147] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:51 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:51.636154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ab3347-654c-4b13-a379-985ae47c19bc None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:51.740835 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:51.749730 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ab3347-654c-4b13-a379-985ae47c19bc None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:51.795509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcbc96a-ff17-4c7b-be7c-1cefc9ed57e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:51.796181 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcbc96a-ff17-4c7b-be7c-1cefc9ed57e6 None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:51.796461 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcbc96a-ff17-4c7b-be7c-1cefc9ed57e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:51.796835 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcbc96a-ff17-4c7b-be7c-1cefc9ed57e6 None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08/users/b4095e120d9b4f6a8a1c912193b8823f/roles/e8be279b370349c8a5b9a06dd763e1b5` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:51.800018 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcbc96a-ff17-4c7b-be7c-1cefc9ed57e6 None admin] RBAC: Authorizing `identity:create_grant(project_id=d6715600d626423b8e8a06d45bb53d08, user_id=b4095e120d9b4f6a8a1c912193b8823f, role_id=e8be279b370349c8a5b9a06dd763e1b5)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:51.801480 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcbc96a-ff17-4c7b-be7c-1cefc9ed57e6 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:51.821634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 73/148] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:46:51 2022] PUT /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08/users/b4095e120d9b4f6a8a1c912193b8823f/roles/e8be279b370349c8a5b9a06dd763e1b5 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:46:52.274635 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1baa0c-bc2d-4412-855d-4b9eaf405993 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:52.275167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1baa0c-bc2d-4412-855d-4b9eaf405993 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:52.275246 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1baa0c-bc2d-4412-855d-4b9eaf405993 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:52.276590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 76/149] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:52.282645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcf8ca0-d677-4b5a-87d0-27e80da97eee None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:52.283109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcf8ca0-d677-4b5a-87d0-27e80da97eee None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:52.283197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcf8ca0-d677-4b5a-87d0-27e80da97eee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:52.311150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-7dcf8ca0-d677-4b5a-87d0-27e80da97eee None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:52.312414 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-7dcf8ca0-d677-4b5a-87d0-27e80da97eee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:52.395774 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 74/150] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:52.748486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e903f6-bcca-4ba7-8157-ea6bfbeaa39b None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:52.748881 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e903f6-bcca-4ba7-8157-ea6bfbeaa39b None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:52.749065 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e903f6-bcca-4ba7-8157-ea6bfbeaa39b None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:52.750759 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 77/151] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:52.763742 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a53f60e-ac95-4ae7-9dd9-24bf31b9acaa None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:52.764072 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a53f60e-ac95-4ae7-9dd9-24bf31b9acaa None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:52.764301 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a53f60e-ac95-4ae7-9dd9-24bf31b9acaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:52.791191 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-0a53f60e-ac95-4ae7-9dd9-24bf31b9acaa None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:52.792398 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-0a53f60e-ac95-4ae7-9dd9-24bf31b9acaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:52.870319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:52.880781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 75/152] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:52.914804 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e903f6-bcca-4ba7-8157-ea6bfbeaa39b None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:53.006215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8ee533-5a47-4fc5-a7ac-8349ff0f974a None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:53.006390 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8ee533-5a47-4fc5-a7ac-8349ff0f974a None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:53.006669 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8ee533-5a47-4fc5-a7ac-8349ff0f974a None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:53.008172 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 76/153] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:46:53.013553 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc07ec83-aa00-4baa-b16a-b578f4e4904d None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:53.013834 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc07ec83-aa00-4baa-b16a-b578f4e4904d None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:53.014107 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc07ec83-aa00-4baa-b16a-b578f4e4904d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:53.040452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-dc07ec83-aa00-4baa-b16a-b578f4e4904d None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:53.041649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-dc07ec83-aa00-4baa-b16a-b578f4e4904d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:53.077764 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e903f6-bcca-4ba7-8157-ea6bfbeaa39b None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:53.142234 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba7e327-82b7-45c9-82e9-d180d1c1eb0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:53.142872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba7e327-82b7-45c9-82e9-d180d1c1eb0e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:53.143156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba7e327-82b7-45c9-82e9-d180d1c1eb0e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:53.143431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba7e327-82b7-45c9-82e9-d180d1c1eb0e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:53.161979 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 77/154] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:53.166753 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba7e327-82b7-45c9-82e9-d180d1c1eb0e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:53.172314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-4ba7e327-82b7-45c9-82e9-d180d1c1eb0e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:46:53.172314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:53.172314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:53.172314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:53.172314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:53.172314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:53.172314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:53.172314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:53.172314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:53.172314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:53.172314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:53.172314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:53.172314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:53.172314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:53.172314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:53.172314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:53.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:53.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:53.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:53.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:53.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:53.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:53.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:53.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:53.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:53.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:53.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:53.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:53.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:53.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:53.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:53.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:53.174089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:53.174089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:53.174089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:53.174089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:53.174089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:53.174089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:53.174089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:53.174089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:53.174089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:53.174089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:53.174089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:46:53.174089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:46:53.174089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 78/155] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:46:52 2022] GET /identity/v3/roles/member => generated 84 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:53.181308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc07ec83-aa00-4baa-b16a-b578f4e4904d None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:53.240345 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba7e327-82b7-45c9-82e9-d180d1c1eb0e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:53.315752 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc07ec83-aa00-4baa-b16a-b578f4e4904d None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:53.316311 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:53.354491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:53.372698 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1ea79e-5417-4c47-9e9f-b5afb225bffe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:53.373333 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1ea79e-5417-4c47-9e9f-b5afb225bffe None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:53.373613 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1ea79e-5417-4c47-9e9f-b5afb225bffe None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:53.373886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1ea79e-5417-4c47-9e9f-b5afb225bffe None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:53.374610 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1ea79e-5417-4c47-9e9f-b5afb225bffe None admin] RBAC: Adding query filter params (name=member) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:53.375487 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1ea79e-5417-4c47-9e9f-b5afb225bffe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:53.376892 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1ea79e-5417-4c47-9e9f-b5afb225bffe None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:53.394910 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 78/156] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:46:53 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:53.400658 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1ea79e-5417-4c47-9e9f-b5afb225bffe None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:53.414297 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba7e327-82b7-45c9-82e9-d180d1c1eb0e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:53.465671 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca75c3a-3331-42a6-b59b-920e8fc4aa7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:53.466341 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca75c3a-3331-42a6-b59b-920e8fc4aa7f None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:53.466620 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca75c3a-3331-42a6-b59b-920e8fc4aa7f None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:53.466889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca75c3a-3331-42a6-b59b-920e8fc4aa7f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:53.483685 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca75c3a-3331-42a6-b59b-920e8fc4aa7f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:53.498270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-bca75c3a-3331-42a6-b59b-920e8fc4aa7f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:46:53.498270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:53.498270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:53.498270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:53.498270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:53.498270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:53.498270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:53.498270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:53.498270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:53.498270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:53.498270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:53.498270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:53.498270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:53.498270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:53.498270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:53.498270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:53.499361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:53.499361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:53.499361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:53.499361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:53.499361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:53.499361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:53.499361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:53.499361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:53.499361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:53.499361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:53.499361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:53.499361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:53.499361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:53.499361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:53.499361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:53.499361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:53.500626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:53.500626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:53.500626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:53.500626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:53.500626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:53.500626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:53.500626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:53.500626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:53.500626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:53.500626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:53.500626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:46:53.500626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:46:53.500626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 79/157] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:46:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:46:53.523362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca75c3a-3331-42a6-b59b-920e8fc4aa7f None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:53.568830 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1ea79e-5417-4c47-9e9f-b5afb225bffe None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:53.618529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b211995f-1c50-4814-85f0-abb1dcb58191 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:53.619237 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b211995f-1c50-4814-85f0-abb1dcb58191 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:53.619520 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b211995f-1c50-4814-85f0-abb1dcb58191 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:53.619792 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b211995f-1c50-4814-85f0-abb1dcb58191 None admin] PATH_INFO: `/v3/users/65508fe717ca4eb996135023e2296dd1` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:53.623229 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b211995f-1c50-4814-85f0-abb1dcb58191 None admin] RBAC: Authorizing `identity:get_user(user_id=65508fe717ca4eb996135023e2296dd1)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:53.624903 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b211995f-1c50-4814-85f0-abb1dcb58191 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:53.626921 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 79/158] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:53 2022] GET /identity/v3/users/65508fe717ca4eb996135023e2296dd1 => generated 275 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:46:53.632724 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b211995f-1c50-4814-85f0-abb1dcb58191 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:53.694162 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca75c3a-3331-42a6-b59b-920e8fc4aa7f None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:53.696739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:53.722487 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:53.749570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a602a1-dffe-47c6-80d0-3132973f58df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:53.750389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a602a1-dffe-47c6-80d0-3132973f58df None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:53.750667 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a602a1-dffe-47c6-80d0-3132973f58df None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:53.751048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a602a1-dffe-47c6-80d0-3132973f58df None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:53.751725 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a602a1-dffe-47c6-80d0-3132973f58df None admin] RBAC: Adding query filter params (name=admin) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:53.752027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a602a1-dffe-47c6-80d0-3132973f58df None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:53.753424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a602a1-dffe-47c6-80d0-3132973f58df None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:53.769397 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 80/159] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:46:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:53.770048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a602a1-dffe-47c6-80d0-3132973f58df None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:53.793250 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b211995f-1c50-4814-85f0-abb1dcb58191 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:53.874393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7678467-6df0-425b-8e21-19133fa4ad84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:53.875586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7678467-6df0-425b-8e21-19133fa4ad84 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:53.875872 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7678467-6df0-425b-8e21-19133fa4ad84 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:53.876155 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7678467-6df0-425b-8e21-19133fa4ad84 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:53.897324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a602a1-dffe-47c6-80d0-3132973f58df None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:53.943591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6357c43d-a715-416d-a930-cc86719abcb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:53.944224 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6357c43d-a715-416d-a930-cc86719abcb2 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:53.944500 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6357c43d-a715-416d-a930-cc86719abcb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:53.944770 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6357c43d-a715-416d-a930-cc86719abcb2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:53.961981 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6357c43d-a715-416d-a930-cc86719abcb2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:53.967535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-6357c43d-a715-416d-a930-cc86719abcb2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:46:53.967535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:53.967535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:53.967535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:53.967535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:53.967535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:53.967535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:53.967535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:53.967535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:53.967535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:53.967535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:53.967535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:53.967535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:53.967535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:53.967535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:53.967535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:53.968447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:53.968447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:53.968447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:53.968447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:53.968447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:53.968447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:53.968447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:53.968447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:53.968447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:53.968447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:53.968447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:53.968447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:53.968447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:53.968447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:53.968447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:53.968447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:53.970178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:53.970178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:53.970178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:53.970178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:53.970178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:53.970178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:53.970178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:53.970178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:53.970178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:53.970178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:53.970178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:46:53.970178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:46:53.970178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 81/160] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:46:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:53.974632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6357c43d-a715-416d-a930-cc86719abcb2 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:54.012171 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7678467-6df0-425b-8e21-19133fa4ad84 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:54.013675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7678467-6df0-425b-8e21-19133fa4ad84 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:54.022462 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:54.129332 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:54.200916 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6357c43d-a715-416d-a930-cc86719abcb2 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:54.201931 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 80/161] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:46:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 570 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:46:54.203860 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcfe092-c3dc-43d8-9922-40cf796b39fb None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:54.204145 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcfe092-c3dc-43d8-9922-40cf796b39fb None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:54.204416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcfe092-c3dc-43d8-9922-40cf796b39fb None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:54.205931 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 81/162] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:54.206697 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcfe092-c3dc-43d8-9922-40cf796b39fb None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:54.284185 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef36067e-4f46-4895-8be3-a7c36c56011e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:54.284827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef36067e-4f46-4895-8be3-a7c36c56011e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:54.285103 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef36067e-4f46-4895-8be3-a7c36c56011e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:54.285373 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef36067e-4f46-4895-8be3-a7c36c56011e None admin] PATH_INFO: `/v3/users/c81176b6299a493294b869b497bad7f4` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:54.290789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef36067e-4f46-4895-8be3-a7c36c56011e None admin] RBAC: Authorizing `identity:get_user(user_id=c81176b6299a493294b869b497bad7f4)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:54.292226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef36067e-4f46-4895-8be3-a7c36c56011e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:54.293998 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 82/163] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:53 2022] GET /identity/v3/users/c81176b6299a493294b869b497bad7f4 => generated 247 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:54.294623 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef36067e-4f46-4895-8be3-a7c36c56011e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:54.446687 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef36067e-4f46-4895-8be3-a7c36c56011e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:54.450566 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:54.466780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcfe092-c3dc-43d8-9922-40cf796b39fb None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:54.525320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69cff93-e21a-4a7f-b7a5-d6e01c2c4325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:54.527029 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69cff93-e21a-4a7f-b7a5-d6e01c2c4325 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:54.527029 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69cff93-e21a-4a7f-b7a5-d6e01c2c4325 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:54.527029 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69cff93-e21a-4a7f-b7a5-d6e01c2c4325 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:54.527241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d6ab70-5497-4f97-9849-b1c0f1f6741e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:54.527348 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69cff93-e21a-4a7f-b7a5-d6e01c2c4325 None admin] RBAC: Adding query filter params (name=member) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:54.527593 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d6ab70-5497-4f97-9849-b1c0f1f6741e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:54.527719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69cff93-e21a-4a7f-b7a5-d6e01c2c4325 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:54.527809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d6ab70-5497-4f97-9849-b1c0f1f6741e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:54.528063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d6ab70-5497-4f97-9849-b1c0f1f6741e None admin] PATH_INFO: `/v3/projects/e6d05dd0b455478c8762a69b412f4a4a` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:54.529093 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69cff93-e21a-4a7f-b7a5-d6e01c2c4325 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:54.534844 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d6ab70-5497-4f97-9849-b1c0f1f6741e None admin] RBAC: Authorizing `identity:get_project(project_id=e6d05dd0b455478c8762a69b412f4a4a)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:54.536312 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d6ab70-5497-4f97-9849-b1c0f1f6741e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:54.537996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 83/164] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:54 2022] GET /identity/v3/projects/e6d05dd0b455478c8762a69b412f4a4a => generated 312 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:54.539208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e342f62a-fc50-485e-b677-615988bbe4ae None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:54.539469 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e342f62a-fc50-485e-b677-615988bbe4ae None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:54.539739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e342f62a-fc50-485e-b677-615988bbe4ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:54.544014 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 82/165] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:46:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:54.545018 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69cff93-e21a-4a7f-b7a5-d6e01c2c4325 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:54.568913 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:54.579607 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-e342f62a-fc50-485e-b677-615988bbe4ae None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:54.580848 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-e342f62a-fc50-485e-b677-615988bbe4ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:54.654682 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69cff93-e21a-4a7f-b7a5-d6e01c2c4325 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:54.700007 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab200a4e-d3aa-4896-82c2-6c620010ee1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:54.700606 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab200a4e-d3aa-4896-82c2-6c620010ee1d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:54.700937 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab200a4e-d3aa-4896-82c2-6c620010ee1d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:54.701222 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab200a4e-d3aa-4896-82c2-6c620010ee1d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:54.711443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 84/166] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:54.712328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e342f62a-fc50-485e-b677-615988bbe4ae None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:54.824921 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab200a4e-d3aa-4896-82c2-6c620010ee1d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:54.826419 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab200a4e-d3aa-4896-82c2-6c620010ee1d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:54.854631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:54.878476 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:54.887428 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e342f62a-fc50-485e-b677-615988bbe4ae None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:54.970980 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672f63e0-fa45-4a7a-8d8d-43391149f53e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:54.971532 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672f63e0-fa45-4a7a-8d8d-43391149f53e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:54.971817 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672f63e0-fa45-4a7a-8d8d-43391149f53e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:54.972093 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672f63e0-fa45-4a7a-8d8d-43391149f53e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:54.972838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672f63e0-fa45-4a7a-8d8d-43391149f53e None admin] RBAC: Adding query filter params (role.id=34235948f8f943a99328516c2171cfff, scope.project.id=e6d05dd0b455478c8762a69b412f4a4a, user.id=65508fe717ca4eb996135023e2296dd1) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:54.973145 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672f63e0-fa45-4a7a-8d8d-43391149f53e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:54.974659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672f63e0-fa45-4a7a-8d8d-43391149f53e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:54.981025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 85/167] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:46:54 2022] GET /identity/v3/role_assignments?user.id=65508fe717ca4eb996135023e2296dd1&scope.project.id=e6d05dd0b455478c8762a69b412f4a4a&role.id=34235948f8f943a99328516c2171cfff => generated 608 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:54.981867 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672f63e0-fa45-4a7a-8d8d-43391149f53e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:54.983107 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 83/168] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:46:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:46:54.991755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab200a4e-d3aa-4896-82c2-6c620010ee1d None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:55.160183 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672f63e0-fa45-4a7a-8d8d-43391149f53e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:55.160641 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab200a4e-d3aa-4896-82c2-6c620010ee1d None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:55.199632 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:55.223532 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53705edd-f534-4812-91a8-f6dc5242f8cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:55.224106 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53705edd-f534-4812-91a8-f6dc5242f8cd None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:55.224388 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53705edd-f534-4812-91a8-f6dc5242f8cd None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:55.224659 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53705edd-f534-4812-91a8-f6dc5242f8cd None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:55.225842 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53705edd-f534-4812-91a8-f6dc5242f8cd None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:55.227511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53705edd-f534-4812-91a8-f6dc5242f8cd None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:55.229547 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 84/169] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:54 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:55.230861 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b431d690-a67c-4740-b8af-72c4a9454e73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:55.231524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b431d690-a67c-4740-b8af-72c4a9454e73 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:55.231809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b431d690-a67c-4740-b8af-72c4a9454e73 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:55.232088 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b431d690-a67c-4740-b8af-72c4a9454e73 None admin] PATH_INFO: `/v3/users/65508fe717ca4eb996135023e2296dd1` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:55.237769 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b431d690-a67c-4740-b8af-72c4a9454e73 None admin] RBAC: Authorizing `identity:get_user(user_id=65508fe717ca4eb996135023e2296dd1)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:55.237769 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b431d690-a67c-4740-b8af-72c4a9454e73 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:55.238953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 86/170] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:54 2022] GET /identity/v3/users/65508fe717ca4eb996135023e2296dd1 => generated 275 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:55.244790 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53705edd-f534-4812-91a8-f6dc5242f8cd None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:55.244949 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b431d690-a67c-4740-b8af-72c4a9454e73 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:55.314564 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:55.400469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53705edd-f534-4812-91a8-f6dc5242f8cd None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:55.410298 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b431d690-a67c-4740-b8af-72c4a9454e73 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:55.459048 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f376c820-e1aa-4fd8-8125-d9205c6ba88a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:55.459754 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f376c820-e1aa-4fd8-8125-d9205c6ba88a None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:55.460020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f376c820-e1aa-4fd8-8125-d9205c6ba88a None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:55.460288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f376c820-e1aa-4fd8-8125-d9205c6ba88a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:55.461082 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f376c820-e1aa-4fd8-8125-d9205c6ba88a None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=d6715600d626423b8e8a06d45bb53d08, user.id=c81176b6299a493294b869b497bad7f4) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:55.461389 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f376c820-e1aa-4fd8-8125-d9205c6ba88a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:55.462933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f376c820-e1aa-4fd8-8125-d9205c6ba88a None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:55.467839 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5407ee1-a690-43dc-88f4-136c8ff60799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:55.469021 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5407ee1-a690-43dc-88f4-136c8ff60799 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:55.469437 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5407ee1-a690-43dc-88f4-136c8ff60799 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:55.469782 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5407ee1-a690-43dc-88f4-136c8ff60799 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:55.470127 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 85/171] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:46:55 2022] GET /identity/v3/role_assignments?user.id=c81176b6299a493294b869b497bad7f4&scope.project.id=d6715600d626423b8e8a06d45bb53d08&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 608 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:55.548876 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f376c820-e1aa-4fd8-8125-d9205c6ba88a None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:55.573917 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5407ee1-a690-43dc-88f4-136c8ff60799 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:55.575451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5407ee1-a690-43dc-88f4-136c8ff60799 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:55.606569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:55.666504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:55.716116 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f376c820-e1aa-4fd8-8125-d9205c6ba88a None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:55.755269 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 87/172] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:46:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 511 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:46:55.760846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5407ee1-a690-43dc-88f4-136c8ff60799 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:55.763508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f38ce9e-9ce0-49e3-a667-d86173e7a82a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:55.764175 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f38ce9e-9ce0-49e3-a667-d86173e7a82a None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:55.764452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f38ce9e-9ce0-49e3-a667-d86173e7a82a None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:55.764724 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f38ce9e-9ce0-49e3-a667-d86173e7a82a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:55.787398 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f38ce9e-9ce0-49e3-a667-d86173e7a82a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:55.793128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-5f38ce9e-9ce0-49e3-a667-d86173e7a82a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:46:55.793128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:55.793128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:55.793128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:55.793128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:55.793128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:55.793128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:55.793128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:55.793128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:55.793128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:55.793128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:55.793128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:55.793128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:55.793128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:55.793128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:55.793128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:55.794078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:55.794078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:55.794078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:55.794078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:55.794078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:55.794078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:55.794078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:55.794078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:55.794078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:55.794078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:55.794078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:55.794078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:55.794078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:55.794078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:55.794078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:55.794078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:55.794873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:55.794873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:55.794873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:55.794873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:55.794873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:55.794873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:55.794873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:55.794873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:55.794873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:55.794873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:55.794873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:46:55.794873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:46:55.794873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 86/173] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:46:55 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:46:55.801542 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f38ce9e-9ce0-49e3-a667-d86173e7a82a None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:55.927689 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5407ee1-a690-43dc-88f4-136c8ff60799 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:55.948920 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f38ce9e-9ce0-49e3-a667-d86173e7a82a None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:55.954101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:55.963998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:55.978831 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f9a6fc-ae47-440e-a5da-dc630c629800 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:55.980101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f9a6fc-ae47-440e-a5da-dc630c629800 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:55.980388 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f9a6fc-ae47-440e-a5da-dc630c629800 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:55.980666 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f9a6fc-ae47-440e-a5da-dc630c629800 None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:55.982276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f9a6fc-ae47-440e-a5da-dc630c629800 None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:55.985178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f9a6fc-ae47-440e-a5da-dc630c629800 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:55.990271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 88/174] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:55 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:46:55.991722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065b3fe4-6a74-46f5-afc7-100a0fbd4969 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:55.992002 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065b3fe4-6a74-46f5-afc7-100a0fbd4969 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:55.992277 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065b3fe4-6a74-46f5-afc7-100a0fbd4969 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:55.993660 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 89/175] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:55.994789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c5e2f2-3b09-48c9-ad27-9fc6dcab9b8e None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:55.996773 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c5e2f2-3b09-48c9-ad27-9fc6dcab9b8e None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:55.996773 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c5e2f2-3b09-48c9-ad27-9fc6dcab9b8e None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:55.996773 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 90/176] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:55 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:55.997447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c5e2f2-3b09-48c9-ad27-9fc6dcab9b8e None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:56.009524 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf4170f-cfa1-4643-88c0-38a957bd2415 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:56.010356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf4170f-cfa1-4643-88c0-38a957bd2415 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:56.010685 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf4170f-cfa1-4643-88c0-38a957bd2415 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:56.010960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf4170f-cfa1-4643-88c0-38a957bd2415 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:56.011706 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf4170f-cfa1-4643-88c0-38a957bd2415 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:56.012798 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf4170f-cfa1-4643-88c0-38a957bd2415 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:56.014360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf4170f-cfa1-4643-88c0-38a957bd2415 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:56.027022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 87/177] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:46:55 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:56.028862 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb98f0c-85f3-44dc-af73-00dafa5ad938 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:56.029154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb98f0c-85f3-44dc-af73-00dafa5ad938 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:56.029435 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb98f0c-85f3-44dc-af73-00dafa5ad938 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:56.054464 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-acb98f0c-85f3-44dc-af73-00dafa5ad938 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:56.055322 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-acb98f0c-85f3-44dc-af73-00dafa5ad938 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:56.147383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c5e2f2-3b09-48c9-ad27-9fc6dcab9b8e None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:56.149159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 88/178] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:56.150319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adf52db-4ad7-46ea-b0ec-6af988e1c3d9 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:56.150678 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adf52db-4ad7-46ea-b0ec-6af988e1c3d9 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:56.151023 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adf52db-4ad7-46ea-b0ec-6af988e1c3d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:56.171836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-5adf52db-4ad7-46ea-b0ec-6af988e1c3d9 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:56.173055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-5adf52db-4ad7-46ea-b0ec-6af988e1c3d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:56.207684 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c202ad3-ff9a-48ea-adb8-56df15984496 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:56.208803 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c202ad3-ff9a-48ea-adb8-56df15984496 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:56.209381 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c202ad3-ff9a-48ea-adb8-56df15984496 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:56.209909 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c202ad3-ff9a-48ea-adb8-56df15984496 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:56.210865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c202ad3-ff9a-48ea-adb8-56df15984496 None admin] RBAC: Adding query filter params (role.id=34235948f8f943a99328516c2171cfff, scope.project.id=d6715600d626423b8e8a06d45bb53d08, user.id=65508fe717ca4eb996135023e2296dd1) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:56.211646 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c202ad3-ff9a-48ea-adb8-56df15984496 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:56.214253 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c202ad3-ff9a-48ea-adb8-56df15984496 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:56.226162 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 91/179] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:46:55 2022] GET /identity/v3/role_assignments?user.id=65508fe717ca4eb996135023e2296dd1&scope.project.id=d6715600d626423b8e8a06d45bb53d08&role.id=34235948f8f943a99328516c2171cfff => generated 608 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:56.226162 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c202ad3-ff9a-48ea-adb8-56df15984496 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:56.263110 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 89/180] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:56.264522 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1cda12-e78f-401d-b8bb-26d3a4197890 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:56.264812 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1cda12-e78f-401d-b8bb-26d3a4197890 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:56.265089 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1cda12-e78f-401d-b8bb-26d3a4197890 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:56.266519 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 90/181] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:56.272392 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f273dc1-0c7e-4f96-9a5e-97917b566afb None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:56.272689 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f273dc1-0c7e-4f96-9a5e-97917b566afb None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:56.272953 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f273dc1-0c7e-4f96-9a5e-97917b566afb None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:56.274773 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 91/182] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:46:56 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:46:56.275863 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e6f99-587b-4c28-8c6f-728c53d2c3c1 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:56.276145 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e6f99-587b-4c28-8c6f-728c53d2c3c1 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:56.276419 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e6f99-587b-4c28-8c6f-728c53d2c3c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:56.297895 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-4f8e6f99-587b-4c28-8c6f-728c53d2c3c1 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:56.299782 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:56.300472 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-4f8e6f99-587b-4c28-8c6f-728c53d2c3c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:56.333732 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:56.368349 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c202ad3-ff9a-48ea-adb8-56df15984496 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:56.384436 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 92/183] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:56.391416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092d89cd-a191-4d23-af4b-07a9756ff451 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:56.391825 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092d89cd-a191-4d23-af4b-07a9756ff451 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:56.392181 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092d89cd-a191-4d23-af4b-07a9756ff451 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:56.412464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d03817-a9ae-41c1-9f31-110cbdc428a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:56.413108 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d03817-a9ae-41c1-9f31-110cbdc428a8 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:56.413390 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d03817-a9ae-41c1-9f31-110cbdc428a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:56.414330 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d03817-a9ae-41c1-9f31-110cbdc428a8 None admin] PATH_INFO: `/v3/users/65508fe717ca4eb996135023e2296dd1` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:56.415518 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d03817-a9ae-41c1-9f31-110cbdc428a8 None admin] RBAC: Authorizing `identity:get_user(user_id=65508fe717ca4eb996135023e2296dd1)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:56.417267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d03817-a9ae-41c1-9f31-110cbdc428a8 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:56.419252 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 92/184] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:56 2022] GET /identity/v3/users/65508fe717ca4eb996135023e2296dd1 => generated 275 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:56.423616 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-092d89cd-a191-4d23-af4b-07a9756ff451 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:46:56.424924 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-092d89cd-a191-4d23-af4b-07a9756ff451 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:46:56.427209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d03817-a9ae-41c1-9f31-110cbdc428a8 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:56.549945 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d03817-a9ae-41c1-9f31-110cbdc428a8 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:56.554072 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 93/185] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:46:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:46:56.571508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092d89cd-a191-4d23-af4b-07a9756ff451 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:56.578875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:56.605376 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa83555-b55e-48fc-a705-a0f2e8313637 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:56.607876 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa83555-b55e-48fc-a705-a0f2e8313637 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:56.609744 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa83555-b55e-48fc-a705-a0f2e8313637 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:56.610916 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa83555-b55e-48fc-a705-a0f2e8313637 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:56.704621 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:56.712345 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa83555-b55e-48fc-a705-a0f2e8313637 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:56.714459 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa83555-b55e-48fc-a705-a0f2e8313637 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:56.717805 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092d89cd-a191-4d23-af4b-07a9756ff451 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:56.762679 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4899b087-38d2-434e-bc36-80fbebaa428e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:56.763561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4899b087-38d2-434e-bc36-80fbebaa428e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:56.764080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4899b087-38d2-434e-bc36-80fbebaa428e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:56.764551 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4899b087-38d2-434e-bc36-80fbebaa428e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:56.777200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4899b087-38d2-434e-bc36-80fbebaa428e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:56.783305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-4899b087-38d2-434e-bc36-80fbebaa428e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:46:56.783305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:56.783305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:56.783305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:56.783305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:56.783305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:56.783305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:56.783305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:56.783305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:56.783305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:56.783305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:56.783305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:56.783305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:56.783305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:56.783305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:56.783305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:56.784196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:56.784196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:56.784196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:56.784196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:56.784196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:56.784196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:56.784196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:56.784196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:56.784196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:56.784196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:56.784196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:56.784196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:56.784196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:56.784196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:56.784196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:56.784196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:56.784963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:56.784963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:56.784963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:56.784963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:56.784963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:56.784963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:56.784963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:56.784963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:56.784963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:56.784963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:56.784963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:46:56.784963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:46:56.784963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 94/186] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:46:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:46:56.785623 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4899b087-38d2-434e-bc36-80fbebaa428e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:56.839076 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 93/187] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:46:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:46:56.839953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa83555-b55e-48fc-a705-a0f2e8313637 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:56.862629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:56.904454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4899b087-38d2-434e-bc36-80fbebaa428e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:56.937652 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5768e849-6514-4ecd-9361-eb5a7af65c9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:56.938323 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5768e849-6514-4ecd-9361-eb5a7af65c9c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:56.938601 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5768e849-6514-4ecd-9361-eb5a7af65c9c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:56.938868 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5768e849-6514-4ecd-9361-eb5a7af65c9c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:56.941359 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa83555-b55e-48fc-a705-a0f2e8313637 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:56.948721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5768e849-6514-4ecd-9361-eb5a7af65c9c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:56.954069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-5768e849-6514-4ecd-9361-eb5a7af65c9c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:46:56.954069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:56.954069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:56.954069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:56.954069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:56.954069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:56.954069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:56.954069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:56.954069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:56.954069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:56.954069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:56.954069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:56.954069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:56.954069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:56.954069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:56.954069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:56.954930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:56.954930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:56.954930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:56.954930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:56.954930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:56.954930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:56.954930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:56.954930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:56.954930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:56.954930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:56.954930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:56.954930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:56.954930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:56.954930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:56.954930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:56.954930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:56.955727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:56.955727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:56.955727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:56.955727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:56.955727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:56.955727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:56.955727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:56.955727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:56.955727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:56.955727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:56.955727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:46:56.955727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:46:56.955727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 95/188] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:46:56 2022] GET /identity/v3/roles/reader => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:56.956381 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5768e849-6514-4ecd-9361-eb5a7af65c9c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:56.972446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:56.981608 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb540eed-1d3d-4edc-97f5-a4b5e400591c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:56.982271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb540eed-1d3d-4edc-97f5-a4b5e400591c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:56.982549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb540eed-1d3d-4edc-97f5-a4b5e400591c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:56.982832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb540eed-1d3d-4edc-97f5-a4b5e400591c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:56.983927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb540eed-1d3d-4edc-97f5-a4b5e400591c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:56.985439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb540eed-1d3d-4edc-97f5-a4b5e400591c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:56.987059 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 94/189] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:46:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:56.990987 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb540eed-1d3d-4edc-97f5-a4b5e400591c None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:57.056072 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5768e849-6514-4ecd-9361-eb5a7af65c9c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:57.076749 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:57.090066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb540eed-1d3d-4edc-97f5-a4b5e400591c None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:57.092421 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da88b7b-acc5-40ea-a0f9-50002c9a014e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:57.093038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da88b7b-acc5-40ea-a0f9-50002c9a014e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:57.093318 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da88b7b-acc5-40ea-a0f9-50002c9a014e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:57.093610 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da88b7b-acc5-40ea-a0f9-50002c9a014e None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:57.094311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da88b7b-acc5-40ea-a0f9-50002c9a014e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:57.094619 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da88b7b-acc5-40ea-a0f9-50002c9a014e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:57.096017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da88b7b-acc5-40ea-a0f9-50002c9a014e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:57.104046 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 96/190] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:46:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:57.104839 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da88b7b-acc5-40ea-a0f9-50002c9a014e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:57.129993 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f898cc-d813-4f00-bd16-2b10a0d32910 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:57.130589 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f898cc-d813-4f00-bd16-2b10a0d32910 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:57.130864 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f898cc-d813-4f00-bd16-2b10a0d32910 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:57.131135 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f898cc-d813-4f00-bd16-2b10a0d32910 None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:57.132312 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f898cc-d813-4f00-bd16-2b10a0d32910 None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:57.133758 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f898cc-d813-4f00-bd16-2b10a0d32910 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:57.135424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 95/191] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:56 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:57.136176 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f898cc-d813-4f00-bd16-2b10a0d32910 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:57.188075 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:57.213714 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da88b7b-acc5-40ea-a0f9-50002c9a014e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:57.240177 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f898cc-d813-4f00-bd16-2b10a0d32910 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:57.247245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc56852c-f9c4-4732-a78c-8f14d9bdebdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:57.247887 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc56852c-f9c4-4732-a78c-8f14d9bdebdc None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:57.248175 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc56852c-f9c4-4732-a78c-8f14d9bdebdc None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:57.248453 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc56852c-f9c4-4732-a78c-8f14d9bdebdc None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:57.259340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc56852c-f9c4-4732-a78c-8f14d9bdebdc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:46:57.264703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-cc56852c-f9c4-4732-a78c-8f14d9bdebdc None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:46:57.264703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:46:57.264703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:46:57.264703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:46:57.264703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:46:57.264703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:46:57.264703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:46:57.264703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:46:57.264703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:46:57.264703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:46:57.264703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:46:57.264703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:46:57.264703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:46:57.264703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:46:57.264703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:46:57.264703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:46:57.266319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:46:57.266319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:46:57.266319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:46:57.266319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:46:57.266319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:46:57.266319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:46:57.266319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:46:57.266319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:46:57.266319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:46:57.266319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:46:57.266319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:46:57.266319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:46:57.266319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:46:57.266319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:46:57.266319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:46:57.266319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:46:57.267174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:46:57.267174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:46:57.267174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:46:57.267174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:46:57.267174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:46:57.267174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:46:57.267174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:46:57.267174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:46:57.267174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:46:57.267174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:46:57.267174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:46:57.267174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:46:57.267174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 97/192] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:46:57 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:46:57.267174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc56852c-f9c4-4732-a78c-8f14d9bdebdc None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:57.279569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4790a29-114b-466b-b5a3-608cfafe251d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:57.280212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4790a29-114b-466b-b5a3-608cfafe251d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:57.280472 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4790a29-114b-466b-b5a3-608cfafe251d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:57.280742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4790a29-114b-466b-b5a3-608cfafe251d None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:57.281438 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4790a29-114b-466b-b5a3-608cfafe251d None admin] RBAC: Adding query filter params (name=reader) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:57.281738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4790a29-114b-466b-b5a3-608cfafe251d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:57.283142 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4790a29-114b-466b-b5a3-608cfafe251d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:57.291594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 96/193] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:46:57 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:57.294854 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4790a29-114b-466b-b5a3-608cfafe251d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:57.314689 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:57.370290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc56852c-f9c4-4732-a78c-8f14d9bdebdc None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:57.395023 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4790a29-114b-466b-b5a3-608cfafe251d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:57.404432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9368225e-9ec5-4901-9957-0c2ad88f45f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:57.405049 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9368225e-9ec5-4901-9957-0c2ad88f45f1 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:57.405337 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9368225e-9ec5-4901-9957-0c2ad88f45f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:57.405618 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9368225e-9ec5-4901-9957-0c2ad88f45f1 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:57.406424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9368225e-9ec5-4901-9957-0c2ad88f45f1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:57.407876 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9368225e-9ec5-4901-9957-0c2ad88f45f1 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:57.415777 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:57.429050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 98/194] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:46:57 2022] POST /identity/v3/users => generated 297 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:46:57.438476 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aca11c2-3dfe-4a7d-8dd0-77c17cf7484f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:57.438476 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aca11c2-3dfe-4a7d-8dd0-77c17cf7484f None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:57.438653 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9368225e-9ec5-4901-9957-0c2ad88f45f1 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:57.438738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aca11c2-3dfe-4a7d-8dd0-77c17cf7484f None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:57.438738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aca11c2-3dfe-4a7d-8dd0-77c17cf7484f None admin] PATH_INFO: `/v3/users/eedddce36de04fcf8d98b5592b562a6f` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:57.458274 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aca11c2-3dfe-4a7d-8dd0-77c17cf7484f None admin] RBAC: Authorizing `identity:get_user(user_id=eedddce36de04fcf8d98b5592b562a6f)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:57.460711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aca11c2-3dfe-4a7d-8dd0-77c17cf7484f None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:57.468272 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 97/195] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:57 2022] GET /identity/v3/users/eedddce36de04fcf8d98b5592b562a6f => generated 283 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:57.468272 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aca11c2-3dfe-4a7d-8dd0-77c17cf7484f None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:57.545790 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9368225e-9ec5-4901-9957-0c2ad88f45f1 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:57.555732 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:57.583200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aca11c2-3dfe-4a7d-8dd0-77c17cf7484f None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:57.588999 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4561267f-c7d5-4218-a9c3-e6660122cbe9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:57.589632 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4561267f-c7d5-4218-a9c3-e6660122cbe9 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:57.589914 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4561267f-c7d5-4218-a9c3-e6660122cbe9 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:57.590216 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4561267f-c7d5-4218-a9c3-e6660122cbe9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:57.590964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4561267f-c7d5-4218-a9c3-e6660122cbe9 None admin] RBAC: Adding query filter params (role.id=285107fb9b2449fe9f1d29f3075a246f, scope.project.id=d6715600d626423b8e8a06d45bb53d08, user.id=65508fe717ca4eb996135023e2296dd1) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:57.591275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4561267f-c7d5-4218-a9c3-e6660122cbe9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:57.592785 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4561267f-c7d5-4218-a9c3-e6660122cbe9 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:57.599922 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 99/196] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:46:57 2022] GET /identity/v3/role_assignments?user.id=65508fe717ca4eb996135023e2296dd1&scope.project.id=d6715600d626423b8e8a06d45bb53d08&role.id=285107fb9b2449fe9f1d29f3075a246f => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:57.600745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4561267f-c7d5-4218-a9c3-e6660122cbe9 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:57.626370 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61adbdf-62f7-47a0-87ad-dbf291038566 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:57.627013 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61adbdf-62f7-47a0-87ad-dbf291038566 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:57.627299 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61adbdf-62f7-47a0-87ad-dbf291038566 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:57.627575 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61adbdf-62f7-47a0-87ad-dbf291038566 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:57.628273 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61adbdf-62f7-47a0-87ad-dbf291038566 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:57.628580 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61adbdf-62f7-47a0-87ad-dbf291038566 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:57.630016 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61adbdf-62f7-47a0-87ad-dbf291038566 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:57.641100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 98/197] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:46:57 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:57.641436 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61adbdf-62f7-47a0-87ad-dbf291038566 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:57.681972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:57.755867 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4561267f-c7d5-4218-a9c3-e6660122cbe9 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:57.787635 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61adbdf-62f7-47a0-87ad-dbf291038566 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:57.802500 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728e6123-a55a-4e98-992c-e11ebf1bc725 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:57.803153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728e6123-a55a-4e98-992c-e11ebf1bc725 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:57.803721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728e6123-a55a-4e98-992c-e11ebf1bc725 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:57.803721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728e6123-a55a-4e98-992c-e11ebf1bc725 None admin] PATH_INFO: `/v3/users/b4095e120d9b4f6a8a1c912193b8823f` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:57.805318 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728e6123-a55a-4e98-992c-e11ebf1bc725 None admin] RBAC: Authorizing `identity:get_user(user_id=b4095e120d9b4f6a8a1c912193b8823f)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:57.806852 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728e6123-a55a-4e98-992c-e11ebf1bc725 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:57.808481 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 100/198] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:57 2022] GET /identity/v3/users/b4095e120d9b4f6a8a1c912193b8823f => generated 289 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:57.815447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728e6123-a55a-4e98-992c-e11ebf1bc725 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:57.827268 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3fcf66-15d7-47ff-aa32-45b92f4909c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:57.828949 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3fcf66-15d7-47ff-aa32-45b92f4909c3 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:57.829230 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3fcf66-15d7-47ff-aa32-45b92f4909c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:57.829501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3fcf66-15d7-47ff-aa32-45b92f4909c3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:57.845235 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:57.921286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3fcf66-15d7-47ff-aa32-45b92f4909c3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:57.923464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3fcf66-15d7-47ff-aa32-45b92f4909c3 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:57.945893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728e6123-a55a-4e98-992c-e11ebf1bc725 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:57.972573 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:57.981710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e7cb06-dd88-4726-b714-c56a9fe93d02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:57.982440 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e7cb06-dd88-4726-b714-c56a9fe93d02 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:57.982922 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e7cb06-dd88-4726-b714-c56a9fe93d02 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:57.983337 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e7cb06-dd88-4726-b714-c56a9fe93d02 None admin] PATH_INFO: `/v3/users/eedddce36de04fcf8d98b5592b562a6f` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:57.984745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e7cb06-dd88-4726-b714-c56a9fe93d02 None admin] RBAC: Authorizing `identity:get_user(user_id=eedddce36de04fcf8d98b5592b562a6f)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:57.986458 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e7cb06-dd88-4726-b714-c56a9fe93d02 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:57.988559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 101/199] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:46:57 2022] GET /identity/v3/users/eedddce36de04fcf8d98b5592b562a6f => generated 283 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:57.989454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e7cb06-dd88-4726-b714-c56a9fe93d02 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:58.051946 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 99/200] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:46:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:46:58.058163 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3fcf66-15d7-47ff-aa32-45b92f4909c3 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:58.100953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:58.110682 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e7cb06-dd88-4726-b714-c56a9fe93d02 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:58.146463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0d856f-5131-4f96-9cfb-5f78609a8c13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:58.147080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0d856f-5131-4f96-9cfb-5f78609a8c13 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:58.147360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0d856f-5131-4f96-9cfb-5f78609a8c13 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:58.147635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0d856f-5131-4f96-9cfb-5f78609a8c13 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:58.166814 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3fcf66-15d7-47ff-aa32-45b92f4909c3 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:58.204610 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf391bce-5826-407c-9188-e97c9fe8fde0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:58.205351 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf391bce-5826-407c-9188-e97c9fe8fde0 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:58.205730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf391bce-5826-407c-9188-e97c9fe8fde0 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:58.206103 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf391bce-5826-407c-9188-e97c9fe8fde0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:58.213948 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:58.228372 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0d856f-5131-4f96-9cfb-5f78609a8c13 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:58.230435 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0d856f-5131-4f96-9cfb-5f78609a8c13 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:58.282016 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf391bce-5826-407c-9188-e97c9fe8fde0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:58.283888 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf391bce-5826-407c-9188-e97c9fe8fde0 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:58.325993 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:58.346049 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 102/201] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:46:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:46:58.346847 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0d856f-5131-4f96-9cfb-5f78609a8c13 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:58.404854 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 100/202] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:46:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:46:58.409933 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf391bce-5826-407c-9188-e97c9fe8fde0 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:58.476393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:58.489989 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0d856f-5131-4f96-9cfb-5f78609a8c13 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:58.525020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf391bce-5826-407c-9188-e97c9fe8fde0 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:58.533413 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0271ebe2-fb34-4241-8000-b5e9634cb4bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:58.534450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0271ebe2-fb34-4241-8000-b5e9634cb4bb None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:58.534933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0271ebe2-fb34-4241-8000-b5e9634cb4bb None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:58.535400 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0271ebe2-fb34-4241-8000-b5e9634cb4bb None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:58.536819 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0271ebe2-fb34-4241-8000-b5e9634cb4bb None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:58.538595 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0271ebe2-fb34-4241-8000-b5e9634cb4bb None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:58.540550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 103/203] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:58 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:58.541795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0271ebe2-fb34-4241-8000-b5e9634cb4bb None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:58.571347 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85c8148-6387-4f76-8680-c665c2a43489 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:58.572607 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85c8148-6387-4f76-8680-c665c2a43489 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:58.573543 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85c8148-6387-4f76-8680-c665c2a43489 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:58.574020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85c8148-6387-4f76-8680-c665c2a43489 None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:58.575672 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85c8148-6387-4f76-8680-c665c2a43489 None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:58.578151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85c8148-6387-4f76-8680-c665c2a43489 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:58.586083 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 101/204] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:58 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:58.586083 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85c8148-6387-4f76-8680-c665c2a43489 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:58.595184 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:58.671468 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0271ebe2-fb34-4241-8000-b5e9634cb4bb None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:58.696329 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85c8148-6387-4f76-8680-c665c2a43489 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:58.711909 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbe9f94-4b75-4c58-b719-ff889344a2f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:58.712824 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbe9f94-4b75-4c58-b719-ff889344a2f6 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:58.713302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbe9f94-4b75-4c58-b719-ff889344a2f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:58.713869 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbe9f94-4b75-4c58-b719-ff889344a2f6 None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:58.715339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbe9f94-4b75-4c58-b719-ff889344a2f6 None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:58.716988 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbe9f94-4b75-4c58-b719-ff889344a2f6 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:58.719506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 104/205] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:46:58 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:58.720617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbe9f94-4b75-4c58-b719-ff889344a2f6 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:58.732816 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f01cd6f-538b-4b9f-b873-75c6250cff0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:58.733645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f01cd6f-538b-4b9f-b873-75c6250cff0a None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:58.733753 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f01cd6f-538b-4b9f-b873-75c6250cff0a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:58.734093 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f01cd6f-538b-4b9f-b873-75c6250cff0a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:58.734990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f01cd6f-538b-4b9f-b873-75c6250cff0a None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54, user.id=eedddce36de04fcf8d98b5592b562a6f) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:58.735286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f01cd6f-538b-4b9f-b873-75c6250cff0a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:58.736702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f01cd6f-538b-4b9f-b873-75c6250cff0a None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:58.742761 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 102/206] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:46:58 2022] GET /identity/v3/role_assignments?user.id=eedddce36de04fcf8d98b5592b562a6f&scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:58.750275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f01cd6f-538b-4b9f-b873-75c6250cff0a None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:46:58.751278 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:58.834971 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbe9f94-4b75-4c58-b719-ff889344a2f6 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:58.873724 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:46:58.883482 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f34e9ff-4def-45dc-b2ee-29f6f89239b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:58.884464 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f34e9ff-4def-45dc-b2ee-29f6f89239b0 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:58.884963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f34e9ff-4def-45dc-b2ee-29f6f89239b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:58.885432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f34e9ff-4def-45dc-b2ee-29f6f89239b0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:58.886425 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f34e9ff-4def-45dc-b2ee-29f6f89239b0 None admin] RBAC: Adding query filter params (role.id=e8be279b370349c8a5b9a06dd763e1b5, scope.project.id=d6715600d626423b8e8a06d45bb53d08, user.id=b4095e120d9b4f6a8a1c912193b8823f) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:58.886936 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f34e9ff-4def-45dc-b2ee-29f6f89239b0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:58.888882 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f34e9ff-4def-45dc-b2ee-29f6f89239b0 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:58.897466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 105/207] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:46:58 2022] GET /identity/v3/role_assignments?user.id=b4095e120d9b4f6a8a1c912193b8823f&scope.project.id=d6715600d626423b8e8a06d45bb53d08&role.id=e8be279b370349c8a5b9a06dd763e1b5 => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:46:58.950144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f01cd6f-538b-4b9f-b873-75c6250cff0a None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:46:58.995621 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a27b9f6-3468-452a-b5e6-35cfc88c5f85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:46:58.996065 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a27b9f6-3468-452a-b5e6-35cfc88c5f85 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:46:58.996343 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a27b9f6-3468-452a-b5e6-35cfc88c5f85 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:46:58.996614 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a27b9f6-3468-452a-b5e6-35cfc88c5f85 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:46:58.997371 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a27b9f6-3468-452a-b5e6-35cfc88c5f85 None admin] RBAC: Adding query filter params (role.id=285107fb9b2449fe9f1d29f3075a246f, scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54, user.id=eedddce36de04fcf8d98b5592b562a6f) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:46:58.997678 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a27b9f6-3468-452a-b5e6-35cfc88c5f85 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:46:58.999581 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a27b9f6-3468-452a-b5e6-35cfc88c5f85 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:46:59.007808 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 103/208] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:46:58 2022] GET /identity/v3/role_assignments?user.id=eedddce36de04fcf8d98b5592b562a6f&scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54&role.id=285107fb9b2449fe9f1d29f3075a246f => generated 259 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:00.059892 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fd050b-733e-4bfc-b5b2-2e9c7ae2f80b None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:00.059892 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fd050b-733e-4bfc-b5b2-2e9c7ae2f80b None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:00.059892 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fd050b-733e-4bfc-b5b2-2e9c7ae2f80b None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:00.059892 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 106/209] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:00.066142 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75a9cde-6b95-4aa1-997d-a0f654dca480 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:00.066557 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75a9cde-6b95-4aa1-997d-a0f654dca480 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:00.066756 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75a9cde-6b95-4aa1-997d-a0f654dca480 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:00.085245 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-a75a9cde-6b95-4aa1-997d-a0f654dca480 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:00.086500 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-a75a9cde-6b95-4aa1-997d-a0f654dca480 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:00.101454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a67e126-d648-4483-bee6-eac30596cd0f None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:00.102559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a67e126-d648-4483-bee6-eac30596cd0f None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:00.102559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a67e126-d648-4483-bee6-eac30596cd0f None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:00.103810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 107/210] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:00.110061 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4689b2-55a5-4c29-9f85-abe4da13b1aa None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:00.110616 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4689b2-55a5-4c29-9f85-abe4da13b1aa None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:00.111097 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4689b2-55a5-4c29-9f85-abe4da13b1aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:00.135129 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-8c4689b2-55a5-4c29-9f85-abe4da13b1aa None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:00.137222 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-8c4689b2-55a5-4c29-9f85-abe4da13b1aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:00.176760 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 104/211] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:00.211246 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 108/212] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:47:00.450939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75a9cde-6b95-4aa1-997d-a0f654dca480 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:00.478944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4689b2-55a5-4c29-9f85-abe4da13b1aa None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:00.504307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:00.530509 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:00.561421 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75a9cde-6b95-4aa1-997d-a0f654dca480 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:00.592336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4689b2-55a5-4c29-9f85-abe4da13b1aa None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:00.596843 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c07353b-407b-4f68-85b0-195b9a6dc360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:00.597479 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c07353b-407b-4f68-85b0-195b9a6dc360 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:00.597754 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c07353b-407b-4f68-85b0-195b9a6dc360 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:00.598022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c07353b-407b-4f68-85b0-195b9a6dc360 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:00.608488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c07353b-407b-4f68-85b0-195b9a6dc360 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:00.613719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-1c07353b-407b-4f68-85b0-195b9a6dc360 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:00.613719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:00.613719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:00.613719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:00.613719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:00.613719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:00.613719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:00.613719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:00.613719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:00.613719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:00.613719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:00.613719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:00.613719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:00.613719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:00.613719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:00.613719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:00.614632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:00.614632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:00.614632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:00.614632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:00.614632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:00.614632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:00.614632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:00.614632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:00.614632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:00.614632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:00.614632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:00.614632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:00.614632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:00.614632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:00.614632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:00.614632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:00.615824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:00.615824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:00.615824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:00.615824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:00.615824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:00.615824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:00.615824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:00.615824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:00.615824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:00.615824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:00.615824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:00.615824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:00.615824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 105/213] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:00.622642 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c07353b-407b-4f68-85b0-195b9a6dc360 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:00.628525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb453bf-bc97-4360-a4fc-5fd91b1c8b7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:00.629168 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb453bf-bc97-4360-a4fc-5fd91b1c8b7d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:00.629450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb453bf-bc97-4360-a4fc-5fd91b1c8b7d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:00.629724 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb453bf-bc97-4360-a4fc-5fd91b1c8b7d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:00.630862 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb453bf-bc97-4360-a4fc-5fd91b1c8b7d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:00.632322 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb453bf-bc97-4360-a4fc-5fd91b1c8b7d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:00.634200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 109/214] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:00.640322 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb453bf-bc97-4360-a4fc-5fd91b1c8b7d None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:00.748706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c07353b-407b-4f68-85b0-195b9a6dc360 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:00.752344 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:00.768222 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb453bf-bc97-4360-a4fc-5fd91b1c8b7d None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:00.775116 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:00.789667 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773cd6c6-23d9-4140-8848-85bdc3a0fe93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:00.790324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773cd6c6-23d9-4140-8848-85bdc3a0fe93 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:00.790600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773cd6c6-23d9-4140-8848-85bdc3a0fe93 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:00.790871 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773cd6c6-23d9-4140-8848-85bdc3a0fe93 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:00.791530 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773cd6c6-23d9-4140-8848-85bdc3a0fe93 None admin] RBAC: Adding query filter params (name=member) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:00.791832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773cd6c6-23d9-4140-8848-85bdc3a0fe93 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:00.793215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773cd6c6-23d9-4140-8848-85bdc3a0fe93 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:00.802398 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 106/215] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:00.807769 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773cd6c6-23d9-4140-8848-85bdc3a0fe93 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:00.814394 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b0381d-3222-4c81-b358-850e4e46f7e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:00.815390 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b0381d-3222-4c81-b358-850e4e46f7e7 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:00.815877 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b0381d-3222-4c81-b358-850e4e46f7e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:00.816358 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b0381d-3222-4c81-b358-850e4e46f7e7 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:00.817645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b0381d-3222-4c81-b358-850e4e46f7e7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:00.819737 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b0381d-3222-4c81-b358-850e4e46f7e7 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:00.854986 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 110/216] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:47:00 2022] POST /identity/v3/users => generated 297 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:47:00.926808 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773cd6c6-23d9-4140-8848-85bdc3a0fe93 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:00.970384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c99b5d-677e-4df9-b284-7b2b4615beea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:00.971398 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c99b5d-677e-4df9-b284-7b2b4615beea None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:00.971895 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c99b5d-677e-4df9-b284-7b2b4615beea None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:00.972378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c99b5d-677e-4df9-b284-7b2b4615beea None admin] PATH_INFO: `/v3/users/b44b593879b941f2a609b6a8865ed8da` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:00.989042 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c99b5d-677e-4df9-b284-7b2b4615beea None admin] RBAC: Authorizing `identity:get_user(user_id=b44b593879b941f2a609b6a8865ed8da)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:00.990870 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c99b5d-677e-4df9-b284-7b2b4615beea None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:00.993135 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 107/217] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:47:00 2022] GET /identity/v3/users/b44b593879b941f2a609b6a8865ed8da => generated 297 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:00.999149 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b0381d-3222-4c81-b358-850e4e46f7e7 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:01.110532 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b0381d-3222-4c81-b358-850e4e46f7e7 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:01.146941 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa10845a-aee0-458b-bbac-640092e6141a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:01.147649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa10845a-aee0-458b-bbac-640092e6141a None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:01.148065 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa10845a-aee0-458b-bbac-640092e6141a None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:01.148338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa10845a-aee0-458b-bbac-640092e6141a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:01.166961 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:01.226357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa10845a-aee0-458b-bbac-640092e6141a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:01.228161 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa10845a-aee0-458b-bbac-640092e6141a None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:01.289935 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a172376-13c5-4e46-9bcb-8eed86600f91 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:01.290509 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a172376-13c5-4e46-9bcb-8eed86600f91 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:01.290509 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a172376-13c5-4e46-9bcb-8eed86600f91 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:01.291850 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 108/218] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:01.297902 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c634f393-01a0-469a-b546-fb6c18f14dd4 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:01.298550 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c634f393-01a0-469a-b546-fb6c18f14dd4 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:01.299424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c634f393-01a0-469a-b546-fb6c18f14dd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:01.327020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-c634f393-01a0-469a-b546-fb6c18f14dd4 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:01.327653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-c634f393-01a0-469a-b546-fb6c18f14dd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:01.350217 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 111/219] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:01.355671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa10845a-aee0-458b-bbac-640092e6141a None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:01.383335 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:01.416394 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:01.425106 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 109/220] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:47:01.481158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa10845a-aee0-458b-bbac-640092e6141a None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:01.521972 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82da6b4a-d3c9-4985-ba58-87db5a461406 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:01.522601 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82da6b4a-d3c9-4985-ba58-87db5a461406 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:01.523189 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82da6b4a-d3c9-4985-ba58-87db5a461406 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:01.523793 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e93721-4ac6-42f2-9baf-ec06dac582d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:01.524446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e93721-4ac6-42f2-9baf-ec06dac582d3 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:01.524721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e93721-4ac6-42f2-9baf-ec06dac582d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:01.524993 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e93721-4ac6-42f2-9baf-ec06dac582d3 None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:01.526008 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 110/221] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:01.526147 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e93721-4ac6-42f2-9baf-ec06dac582d3 None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:01.530136 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e93721-4ac6-42f2-9baf-ec06dac582d3 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:01.530136 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 112/222] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:01 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:01.534831 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af8638-b707-49f6-88c8-3211ea3a5069 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:01.535348 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af8638-b707-49f6-88c8-3211ea3a5069 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:01.535846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af8638-b707-49f6-88c8-3211ea3a5069 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:01.536717 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e93721-4ac6-42f2-9baf-ec06dac582d3 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:01.571287 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-e2af8638-b707-49f6-88c8-3211ea3a5069 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:01.572868 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-e2af8638-b707-49f6-88c8-3211ea3a5069 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:01.667337 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e93721-4ac6-42f2-9baf-ec06dac582d3 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:01.695993 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:01.704321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 111/223] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:47:01.734483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656abe1a-53c1-42a1-92bc-c64fdcf1ef78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:01.735158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656abe1a-53c1-42a1-92bc-c64fdcf1ef78 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:01.735437 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656abe1a-53c1-42a1-92bc-c64fdcf1ef78 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:01.735707 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656abe1a-53c1-42a1-92bc-c64fdcf1ef78 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:01.736441 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656abe1a-53c1-42a1-92bc-c64fdcf1ef78 None admin] RBAC: Adding query filter params (role.id=34235948f8f943a99328516c2171cfff, scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54, user.id=b44b593879b941f2a609b6a8865ed8da) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:01.736746 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656abe1a-53c1-42a1-92bc-c64fdcf1ef78 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:01.738298 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656abe1a-53c1-42a1-92bc-c64fdcf1ef78 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:01.744317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 113/224] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:47:01 2022] GET /identity/v3/role_assignments?user.id=b44b593879b941f2a609b6a8865ed8da&scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54&role.id=34235948f8f943a99328516c2171cfff => generated 259 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:01.765603 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2af8638-b707-49f6-88c8-3211ea3a5069 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:01.891895 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2af8638-b707-49f6-88c8-3211ea3a5069 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:01.894560 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:01.945751 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4a9dd3-28fd-487d-b6f6-5a8011b01f03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:01.946449 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4a9dd3-28fd-487d-b6f6-5a8011b01f03 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:01.946727 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4a9dd3-28fd-487d-b6f6-5a8011b01f03 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:01.946997 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4a9dd3-28fd-487d-b6f6-5a8011b01f03 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:01.963941 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4a9dd3-28fd-487d-b6f6-5a8011b01f03 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:01.969367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-bd4a9dd3-28fd-487d-b6f6-5a8011b01f03 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:47:01.969367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:01.969367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:01.969367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:01.969367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:01.969367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:01.969367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:01.969367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:01.969367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:01.969367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:01.969367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:01.969367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:01.969367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:01.969367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:01.969367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:01.969367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:01.970336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:01.970336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:01.970336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:01.970336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:01.970336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:01.970336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:01.970336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:01.970336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:01.970336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:01.970336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:01.970336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:01.970336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:01.970336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:01.970336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:01.970336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:01.970336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:01.971122 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:01.971122 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:01.971122 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:01.971122 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:01.971122 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:01.971122 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:01.971122 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:01.971122 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:01.971122 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:01.971122 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:01.971122 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:47:01.971122 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:01.975205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 112/225] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:47:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:01.978687 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656abe1a-53c1-42a1-92bc-c64fdcf1ef78 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:02.080904 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4a9dd3-28fd-487d-b6f6-5a8011b01f03 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:02.154409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656abe1a-53c1-42a1-92bc-c64fdcf1ef78 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:02.193098 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99fec8d-069c-426f-8ad4-e86aabe9495c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:02.193719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99fec8d-069c-426f-8ad4-e86aabe9495c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:02.194002 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99fec8d-069c-426f-8ad4-e86aabe9495c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:02.194307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99fec8d-069c-426f-8ad4-e86aabe9495c None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:02.195441 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99fec8d-069c-426f-8ad4-e86aabe9495c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:02.195752 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99fec8d-069c-426f-8ad4-e86aabe9495c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:02.197468 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99fec8d-069c-426f-8ad4-e86aabe9495c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:02.211603 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 114/226] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:02.215264 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99fec8d-069c-426f-8ad4-e86aabe9495c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:02.274775 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4a9dd3-28fd-487d-b6f6-5a8011b01f03 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:02.316327 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:02.354841 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5c8a04-b30c-4ee4-a3eb-c7a859a09d58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:02.356087 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5c8a04-b30c-4ee4-a3eb-c7a859a09d58 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:02.356515 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5c8a04-b30c-4ee4-a3eb-c7a859a09d58 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:02.356931 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5c8a04-b30c-4ee4-a3eb-c7a859a09d58 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:02.374217 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5c8a04-b30c-4ee4-a3eb-c7a859a09d58 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:02.379865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-0d5c8a04-b30c-4ee4-a3eb-c7a859a09d58 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:02.379865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:02.379865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:02.379865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:02.379865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:02.379865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:02.379865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:02.379865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:02.379865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:02.379865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:02.379865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:02.379865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:02.379865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:02.379865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:02.379865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:02.379865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:02.380745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:02.380745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:02.380745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:02.380745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:02.380745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:02.380745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:02.380745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:02.380745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:02.380745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:02.380745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:02.380745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:02.380745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:02.380745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:02.380745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:02.380745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:02.380745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:02.381547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:02.381547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:02.381547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:02.381547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:02.381547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:02.381547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:02.381547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:02.381547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:02.381547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:02.381547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:02.381547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:02.381547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:02.381547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 113/227] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:47:02 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:02.386868 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5c8a04-b30c-4ee4-a3eb-c7a859a09d58 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:02.413268 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99fec8d-069c-426f-8ad4-e86aabe9495c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:02.414346 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:02.487713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3810a56d-785d-4e59-8ded-76daa98eb9c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:02.488671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3810a56d-785d-4e59-8ded-76daa98eb9c3 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:02.489137 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3810a56d-785d-4e59-8ded-76daa98eb9c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:02.489593 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3810a56d-785d-4e59-8ded-76daa98eb9c3 None admin] PATH_INFO: `/v3/users/eedddce36de04fcf8d98b5592b562a6f` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:02.490949 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3810a56d-785d-4e59-8ded-76daa98eb9c3 None admin] RBAC: Authorizing `identity:get_user(user_id=eedddce36de04fcf8d98b5592b562a6f)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:02.492624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3810a56d-785d-4e59-8ded-76daa98eb9c3 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:02.494581 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 115/228] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:47:02 2022] GET /identity/v3/users/eedddce36de04fcf8d98b5592b562a6f => generated 283 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:02.498156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5c8a04-b30c-4ee4-a3eb-c7a859a09d58 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:02.500606 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3810a56d-785d-4e59-8ded-76daa98eb9c3 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:02.534180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f823763-0865-425b-bc20-6995342b4d69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:02.534859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f823763-0865-425b-bc20-6995342b4d69 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:02.535143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f823763-0865-425b-bc20-6995342b4d69 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:02.535417 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f823763-0865-425b-bc20-6995342b4d69 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:02.536147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f823763-0865-425b-bc20-6995342b4d69 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:02.536504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f823763-0865-425b-bc20-6995342b4d69 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:02.537903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f823763-0865-425b-bc20-6995342b4d69 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:02.547700 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 114/229] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:02 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:02.553320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f823763-0865-425b-bc20-6995342b4d69 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:02.645743 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:02.662435 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f823763-0865-425b-bc20-6995342b4d69 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:02.704188 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d9ba1a-c540-4223-a92c-4c95852062a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:02.704733 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d9ba1a-c540-4223-a92c-4c95852062a6 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:02.705014 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d9ba1a-c540-4223-a92c-4c95852062a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:02.705287 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d9ba1a-c540-4223-a92c-4c95852062a6 None admin] PATH_INFO: `/v3/users/eedddce36de04fcf8d98b5592b562a6f` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:02.706890 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d9ba1a-c540-4223-a92c-4c95852062a6 None admin] RBAC: Authorizing `identity:get_user(user_id=eedddce36de04fcf8d98b5592b562a6f)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:02.708339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d9ba1a-c540-4223-a92c-4c95852062a6 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:02.710183 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 115/230] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:47:02 2022] GET /identity/v3/users/eedddce36de04fcf8d98b5592b562a6f => generated 283 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:02.720077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d9ba1a-c540-4223-a92c-4c95852062a6 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:02.743119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3810a56d-785d-4e59-8ded-76daa98eb9c3 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:02.762470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:02.808704 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c087d883-179f-48e4-847e-5b1fd7f82c9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:02.809356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c087d883-179f-48e4-847e-5b1fd7f82c9c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:02.809638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c087d883-179f-48e4-847e-5b1fd7f82c9c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:02.809919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c087d883-179f-48e4-847e-5b1fd7f82c9c None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:02.811108 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c087d883-179f-48e4-847e-5b1fd7f82c9c None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:02.812843 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c087d883-179f-48e4-847e-5b1fd7f82c9c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:02.814904 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 116/231] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:02 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:02.827210 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c087d883-179f-48e4-847e-5b1fd7f82c9c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:02.881782 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d9ba1a-c540-4223-a92c-4c95852062a6 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:02.923203 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b0651c-17a5-4b90-94cf-3e73f63e13e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:02.923839 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b0651c-17a5-4b90-94cf-3e73f63e13e1 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:02.924120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b0651c-17a5-4b90-94cf-3e73f63e13e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:02.924390 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b0651c-17a5-4b90-94cf-3e73f63e13e1 None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:02.932618 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b0651c-17a5-4b90-94cf-3e73f63e13e1 None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:02.934308 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b0651c-17a5-4b90-94cf-3e73f63e13e1 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:02.936437 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 116/232] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:02 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:02.942851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b0651c-17a5-4b90-94cf-3e73f63e13e1 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:02.977174 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:03.065701 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b0651c-17a5-4b90-94cf-3e73f63e13e1 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:03.100886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c087d883-179f-48e4-847e-5b1fd7f82c9c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:03.109701 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cae8f6-d53b-4447-a8be-2494419155a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:03.110551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cae8f6-d53b-4447-a8be-2494419155a8 None admin] REQUEST_METHOD: `PUT` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:03.110833 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cae8f6-d53b-4447-a8be-2494419155a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:03.111112 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cae8f6-d53b-4447-a8be-2494419155a8 None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54/users/eedddce36de04fcf8d98b5592b562a6f/roles/285107fb9b2449fe9f1d29f3075a246f` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:03.113341 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cae8f6-d53b-4447-a8be-2494419155a8 None admin] RBAC: Authorizing `identity:create_grant(project_id=72fb26d349cf4298b0ad385cd5e2ee54, user_id=eedddce36de04fcf8d98b5592b562a6f, role_id=285107fb9b2449fe9f1d29f3075a246f)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:03.114852 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cae8f6-d53b-4447-a8be-2494419155a8 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:03.142713 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 117/233] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:47:02 2022] PUT /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54/users/eedddce36de04fcf8d98b5592b562a6f/roles/285107fb9b2449fe9f1d29f3075a246f => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 07 03:47:03.153958 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3f78eb-3fba-45ee-a89e-b2a7100ee9e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:03.155045 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3f78eb-3fba-45ee-a89e-b2a7100ee9e4 None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:03.156705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3f78eb-3fba-45ee-a89e-b2a7100ee9e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:03.157533 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3f78eb-3fba-45ee-a89e-b2a7100ee9e4 None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54/users/eedddce36de04fcf8d98b5592b562a6f/roles/9d5b19a61c46475d8048b1bd66ca25a1` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:03.163784 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3f78eb-3fba-45ee-a89e-b2a7100ee9e4 None admin] RBAC: Authorizing `identity:create_grant(project_id=72fb26d349cf4298b0ad385cd5e2ee54, user_id=eedddce36de04fcf8d98b5592b562a6f, role_id=9d5b19a61c46475d8048b1bd66ca25a1)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:03.165626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3f78eb-3fba-45ee-a89e-b2a7100ee9e4 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:03.192489 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 117/234] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:47:02 2022] PUT /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54/users/eedddce36de04fcf8d98b5592b562a6f/roles/9d5b19a61c46475d8048b1bd66ca25a1 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:47:03.737201 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6dbaa8-a121-4cb9-9cab-69fc1ff8a980 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:03.738260 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6dbaa8-a121-4cb9-9cab-69fc1ff8a980 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:03.738742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6dbaa8-a121-4cb9-9cab-69fc1ff8a980 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:03.740546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 118/235] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:03.746490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e340b4f2-97e3-4bec-aed9-5fe83f36258e None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:03.746775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e340b4f2-97e3-4bec-aed9-5fe83f36258e None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:03.747049 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e340b4f2-97e3-4bec-aed9-5fe83f36258e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:03.775187 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-e340b4f2-97e3-4bec-aed9-5fe83f36258e None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:03.776412 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-e340b4f2-97e3-4bec-aed9-5fe83f36258e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:03.810896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:03.870734 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 118/236] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:03.958171 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9d7adc-3252-4114-9279-4c57cde621e2 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:03.958792 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9d7adc-3252-4114-9279-4c57cde621e2 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:03.959261 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9d7adc-3252-4114-9279-4c57cde621e2 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:03.960799 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 119/237] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:03.965985 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d39312b-76ab-4129-9e38-dd5ce2a449c6 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:03.966301 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d39312b-76ab-4129-9e38-dd5ce2a449c6 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:03.966579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d39312b-76ab-4129-9e38-dd5ce2a449c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:03.995166 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-2d39312b-76ab-4129-9e38-dd5ce2a449c6 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:03.996288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-2d39312b-76ab-4129-9e38-dd5ce2a449c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:04.087940 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 119/238] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:04.209653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff151ed-ea31-4751-9af0-33407d440c31 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:04.209653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff151ed-ea31-4751-9af0-33407d440c31 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:04.209653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff151ed-ea31-4751-9af0-33407d440c31 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:04.209653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 120/239] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:04.215019 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fee949-20fb-4e72-8ff1-53d67f06f861 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:04.215449 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fee949-20fb-4e72-8ff1-53d67f06f861 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:04.215634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fee949-20fb-4e72-8ff1-53d67f06f861 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:04.239245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-45fee949-20fb-4e72-8ff1-53d67f06f861 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:04.240456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-45fee949-20fb-4e72-8ff1-53d67f06f861 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:04.341845 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 120/240] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:04.359393 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff151ed-ea31-4751-9af0-33407d440c31 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:04.429999 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fee949-20fb-4e72-8ff1-53d67f06f861 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:04.474491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:04.509753 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:04.542760 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff151ed-ea31-4751-9af0-33407d440c31 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:04.597445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fee949-20fb-4e72-8ff1-53d67f06f861 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:04.603704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d952a46-9134-4e38-a378-e2fbf3ccd5e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:04.604351 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d952a46-9134-4e38-a378-e2fbf3ccd5e7 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:04.604630 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d952a46-9134-4e38-a378-e2fbf3ccd5e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:04.604905 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d952a46-9134-4e38-a378-e2fbf3ccd5e7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:04.606023 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d952a46-9134-4e38-a378-e2fbf3ccd5e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:04.607500 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d952a46-9134-4e38-a378-e2fbf3ccd5e7 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:04.609117 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 121/241] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:04.614736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d952a46-9134-4e38-a378-e2fbf3ccd5e7 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:04.657461 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6eac4e-c2be-4d25-9f06-a72f4bf85c98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:04.658132 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6eac4e-c2be-4d25-9f06-a72f4bf85c98 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:04.658663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6eac4e-c2be-4d25-9f06-a72f4bf85c98 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:04.659107 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6eac4e-c2be-4d25-9f06-a72f4bf85c98 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:04.679885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6eac4e-c2be-4d25-9f06-a72f4bf85c98 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:04.686004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-9b6eac4e-c2be-4d25-9f06-a72f4bf85c98 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:47:04.686004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:04.686004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:04.686004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:04.686004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:04.686004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:04.686004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:04.686004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:04.686004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:04.686004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:04.686004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:04.686004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:04.686004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:04.686004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:04.686004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:04.686004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:04.687745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:04.687745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:04.687745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:04.687745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:04.687745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:04.687745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:04.687745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:04.687745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:04.687745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:04.687745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:04.687745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:04.687745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:04.687745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:04.687745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:04.687745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:04.687745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:04.688638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:04.688638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:04.688638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:04.688638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:04.688638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:04.688638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:04.688638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:04.688638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:04.688638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:04.688638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:04.688638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:47:04.688638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:04.695185 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 121/242] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:04 2022] GET /identity/v3/roles/reader => generated 84 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:04.701642 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6eac4e-c2be-4d25-9f06-a72f4bf85c98 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:04.778389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d952a46-9134-4e38-a378-e2fbf3ccd5e7 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:04.815005 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:04.819712 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b040eb6-74ec-4264-9c4a-6b5854db8cd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:04.820631 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b040eb6-74ec-4264-9c4a-6b5854db8cd9 None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:04.820912 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b040eb6-74ec-4264-9c4a-6b5854db8cd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:04.821183 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b040eb6-74ec-4264-9c4a-6b5854db8cd9 None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:04.821897 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b040eb6-74ec-4264-9c4a-6b5854db8cd9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:04.823534 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b040eb6-74ec-4264-9c4a-6b5854db8cd9 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:04.833812 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6eac4e-c2be-4d25-9f06-a72f4bf85c98 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:04.847795 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 122/243] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:47:04 2022] POST /identity/v3/users => generated 293 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 07 03:47:04.854142 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b040eb6-74ec-4264-9c4a-6b5854db8cd9 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:04.861099 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:04.899394 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b247090a-d57e-4c5e-83a3-d47164215252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:04.900318 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b247090a-d57e-4c5e-83a3-d47164215252 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:04.900748 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b247090a-d57e-4c5e-83a3-d47164215252 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:04.901167 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b247090a-d57e-4c5e-83a3-d47164215252 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:04.901994 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b247090a-d57e-4c5e-83a3-d47164215252 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:04.902463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b247090a-d57e-4c5e-83a3-d47164215252 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:04.904046 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b247090a-d57e-4c5e-83a3-d47164215252 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:04.922524 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 122/244] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:04 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:04.935194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b247090a-d57e-4c5e-83a3-d47164215252 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:04.978159 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b040eb6-74ec-4264-9c4a-6b5854db8cd9 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:05.016422 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e85f993-eac9-464b-b08d-ea144a9494fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:05.017180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e85f993-eac9-464b-b08d-ea144a9494fd None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:05.017359 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e85f993-eac9-464b-b08d-ea144a9494fd None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:05.017635 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e85f993-eac9-464b-b08d-ea144a9494fd None admin] PATH_INFO: `/v3/roles/member` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:05.034836 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e85f993-eac9-464b-b08d-ea144a9494fd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:05.035022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-9e85f993-eac9-464b-b08d-ea144a9494fd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:05.035022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:05.035022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:05.035022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:05.035022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:05.035022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:05.035022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:05.035022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:05.035022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:05.035022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:05.035022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:05.035022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:05.035022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:05.035022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:05.035022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:05.035022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:05.036034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:05.036034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:05.036034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:05.036034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:05.036034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:05.036034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:05.036034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:05.036034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:05.036034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:05.036034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:05.036034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:05.036034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:05.036034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:05.036034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:05.036034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:05.036034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:05.036989 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:05.036989 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:05.036989 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:05.036989 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:05.036989 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:05.036989 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:05.036989 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:05.036989 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:05.036989 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:05.036989 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:05.036989 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:05.036989 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:05.036989 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 123/245] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:05.042478 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e85f993-eac9-464b-b08d-ea144a9494fd None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:05.089131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b247090a-d57e-4c5e-83a3-d47164215252 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:05.125289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b08beb-3fa6-4229-bffd-cab93fe16d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:05.126416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b08beb-3fa6-4229-bffd-cab93fe16d4f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:05.126884 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b08beb-3fa6-4229-bffd-cab93fe16d4f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:05.127340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b08beb-3fa6-4229-bffd-cab93fe16d4f None admin] PATH_INFO: `/v3/users/8c293eb6aff84de19d8379d626cdde03` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:05.143100 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b08beb-3fa6-4229-bffd-cab93fe16d4f None admin] RBAC: Authorizing `identity:get_user(user_id=8c293eb6aff84de19d8379d626cdde03)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:05.144886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b08beb-3fa6-4229-bffd-cab93fe16d4f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:05.147207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 123/246] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:47:04 2022] GET /identity/v3/users/8c293eb6aff84de19d8379d626cdde03 => generated 297 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:05.152691 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b08beb-3fa6-4229-bffd-cab93fe16d4f None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:05.162053 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:05.190598 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:05.247137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e85f993-eac9-464b-b08d-ea144a9494fd None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:05.285494 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a718ac-ebae-4624-9769-e1ad4f13abd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:05.286107 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a718ac-ebae-4624-9769-e1ad4f13abd1 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:05.286428 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a718ac-ebae-4624-9769-e1ad4f13abd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:05.286696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a718ac-ebae-4624-9769-e1ad4f13abd1 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:05.287504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b08beb-3fa6-4229-bffd-cab93fe16d4f None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:05.288336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a718ac-ebae-4624-9769-e1ad4f13abd1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:05.288803 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a718ac-ebae-4624-9769-e1ad4f13abd1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:05.290468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a718ac-ebae-4624-9769-e1ad4f13abd1 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:05.300529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 124/247] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:05.306313 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a718ac-ebae-4624-9769-e1ad4f13abd1 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:05.325805 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9506e6c5-6731-43ec-aebc-de7bf7925681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:05.326657 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9506e6c5-6731-43ec-aebc-de7bf7925681 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:05.327000 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9506e6c5-6731-43ec-aebc-de7bf7925681 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:05.327378 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9506e6c5-6731-43ec-aebc-de7bf7925681 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:05.403759 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9506e6c5-6731-43ec-aebc-de7bf7925681 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:05.405249 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9506e6c5-6731-43ec-aebc-de7bf7925681 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:05.418459 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:05.433312 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a718ac-ebae-4624-9769-e1ad4f13abd1 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:05.472959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:05.484229 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825d4bd5-e720-483c-a17b-a5afc715add5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:05.484882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825d4bd5-e720-483c-a17b-a5afc715add5 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:05.485468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825d4bd5-e720-483c-a17b-a5afc715add5 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:05.485750 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825d4bd5-e720-483c-a17b-a5afc715add5 None admin] PATH_INFO: `/v3/users/b44b593879b941f2a609b6a8865ed8da` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:05.487113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825d4bd5-e720-483c-a17b-a5afc715add5 None admin] RBAC: Authorizing `identity:get_user(user_id=b44b593879b941f2a609b6a8865ed8da)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:05.489875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825d4bd5-e720-483c-a17b-a5afc715add5 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:05.491173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 125/248] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:47:05 2022] GET /identity/v3/users/b44b593879b941f2a609b6a8865ed8da => generated 297 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:05.496808 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825d4bd5-e720-483c-a17b-a5afc715add5 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:05.536983 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 124/249] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:05.544971 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9506e6c5-6731-43ec-aebc-de7bf7925681 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:05.652321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825d4bd5-e720-483c-a17b-a5afc715add5 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:05.695999 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a12817-f936-4214-961e-cda192fa8ad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:05.696537 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a12817-f936-4214-961e-cda192fa8ad6 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:05.696836 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a12817-f936-4214-961e-cda192fa8ad6 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:05.697912 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a12817-f936-4214-961e-cda192fa8ad6 None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:05.699178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a12817-f936-4214-961e-cda192fa8ad6 None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:05.700622 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a12817-f936-4214-961e-cda192fa8ad6 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:05.703731 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 126/250] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:05 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:05.704724 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9506e6c5-6731-43ec-aebc-de7bf7925681 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:05.707515 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:05.709881 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a12817-f936-4214-961e-cda192fa8ad6 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:05.752285 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11407b7-f8b8-4813-a7f8-aa7a59b4598f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:05.754265 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11407b7-f8b8-4813-a7f8-aa7a59b4598f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:05.754837 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11407b7-f8b8-4813-a7f8-aa7a59b4598f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:05.755318 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11407b7-f8b8-4813-a7f8-aa7a59b4598f None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:05.756697 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11407b7-f8b8-4813-a7f8-aa7a59b4598f None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:05.758412 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11407b7-f8b8-4813-a7f8-aa7a59b4598f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:05.761868 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 125/251] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:05 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:05.770921 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11407b7-f8b8-4813-a7f8-aa7a59b4598f None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:05.795994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:05.846778 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a12817-f936-4214-961e-cda192fa8ad6 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:05.883563 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e143244-1e21-4016-af9e-215a00211005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:05.884237 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e143244-1e21-4016-af9e-215a00211005 None admin] REQUEST_METHOD: `PUT` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:05.884514 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e143244-1e21-4016-af9e-215a00211005 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:05.884790 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e143244-1e21-4016-af9e-215a00211005 None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54/users/b44b593879b941f2a609b6a8865ed8da/roles/34235948f8f943a99328516c2171cfff` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:05.886764 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e143244-1e21-4016-af9e-215a00211005 None admin] RBAC: Authorizing `identity:create_grant(project_id=72fb26d349cf4298b0ad385cd5e2ee54, user_id=b44b593879b941f2a609b6a8865ed8da, role_id=34235948f8f943a99328516c2171cfff)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:05.888273 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e143244-1e21-4016-af9e-215a00211005 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:05.898941 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11407b7-f8b8-4813-a7f8-aa7a59b4598f None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:05.904134 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 127/252] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:47:05 2022] PUT /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54/users/b44b593879b941f2a609b6a8865ed8da/roles/34235948f8f943a99328516c2171cfff => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:47:05.905453 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f68ac2-ff0c-4d3c-b91b-ae9d002a0f30 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:05.910553 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f68ac2-ff0c-4d3c-b91b-ae9d002a0f30 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:05.910947 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f68ac2-ff0c-4d3c-b91b-ae9d002a0f30 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:05.912584 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 128/253] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:05 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:05.918750 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c1e407-5d01-4151-ad99-1d59e2752c70 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:05.919042 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c1e407-5d01-4151-ad99-1d59e2752c70 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:05.919379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c1e407-5d01-4151-ad99-1d59e2752c70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:05.941807 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad21183a-b556-4e9b-8443-80e575816ac1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:05.942994 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad21183a-b556-4e9b-8443-80e575816ac1 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:05.943344 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad21183a-b556-4e9b-8443-80e575816ac1 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:05.943695 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad21183a-b556-4e9b-8443-80e575816ac1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:05.944525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad21183a-b556-4e9b-8443-80e575816ac1 None admin] RBAC: Adding query filter params (role.id=e8be279b370349c8a5b9a06dd763e1b5, scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54, user.id=8c293eb6aff84de19d8379d626cdde03) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:05.944904 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad21183a-b556-4e9b-8443-80e575816ac1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:05.946480 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad21183a-b556-4e9b-8443-80e575816ac1 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:05.950742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-23c1e407-5d01-4151-ad99-1d59e2752c70 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:05.953320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-23c1e407-5d01-4151-ad99-1d59e2752c70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:05.954335 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 126/254] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:47:05 2022] GET /identity/v3/role_assignments?user.id=8c293eb6aff84de19d8379d626cdde03&scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54&role.id=e8be279b370349c8a5b9a06dd763e1b5 => generated 259 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:05.955780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b62ad82-7f0f-4de2-b309-d5baab320b23 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:05.956133 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b62ad82-7f0f-4de2-b309-d5baab320b23 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:05.956482 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b62ad82-7f0f-4de2-b309-d5baab320b23 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:05.958154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 127/255] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:05.964628 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e9e5c4-a6bb-43a9-b03a-f443645a7ca2 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:05.965235 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e9e5c4-a6bb-43a9-b03a-f443645a7ca2 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:05.965714 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e9e5c4-a6bb-43a9-b03a-f443645a7ca2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:05.986645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-56e9e5c4-a6bb-43a9-b03a-f443645a7ca2 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:05.988245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-56e9e5c4-a6bb-43a9-b03a-f443645a7ca2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:06.077160 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:06.119841 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 129/256] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:47:06.160476 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 128/257] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:06.451188 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c1e407-5d01-4151-ad99-1d59e2752c70 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:06.524757 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:06.576528 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c1e407-5d01-4151-ad99-1d59e2752c70 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:06.612800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b3e3b6-0e44-428b-bdaa-40571323f0e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:06.613448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b3e3b6-0e44-428b-bdaa-40571323f0e6 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:06.613725 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b3e3b6-0e44-428b-bdaa-40571323f0e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:06.613993 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b3e3b6-0e44-428b-bdaa-40571323f0e6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:06.625448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b3e3b6-0e44-428b-bdaa-40571323f0e6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:06.630702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-d2b3e3b6-0e44-428b-bdaa-40571323f0e6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:06.630702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:06.630702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:06.630702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:06.630702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:06.630702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:06.630702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:06.630702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:06.630702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:06.630702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:06.630702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:06.630702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:06.630702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:06.630702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:06.630702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:06.630702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:06.631582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:06.631582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:06.631582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:06.631582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:06.631582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:06.631582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:06.631582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:06.631582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:06.631582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:06.631582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:06.631582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:06.631582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:06.631582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:06.631582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:06.631582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:06.631582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:06.632357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:06.632357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:06.632357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:06.632357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:06.632357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:06.632357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:06.632357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:06.632357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:06.632357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:06.632357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:06.632357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:06.632357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:06.632357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 130/258] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:47:06 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:06.639783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e9e5c4-a6bb-43a9-b03a-f443645a7ca2 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:06.674986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b3e3b6-0e44-428b-bdaa-40571323f0e6 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:06.758252 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e9e5c4-a6bb-43a9-b03a-f443645a7ca2 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:06.850930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096d8ec5-9375-480d-adaf-4bc10d8757ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:06.851984 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d8ec5-9375-480d-adaf-4bc10d8757ff None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:06.852550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d8ec5-9375-480d-adaf-4bc10d8757ff None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:06.853046 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d8ec5-9375-480d-adaf-4bc10d8757ff None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:06.853961 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096d8ec5-9375-480d-adaf-4bc10d8757ff None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:06.854538 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096d8ec5-9375-480d-adaf-4bc10d8757ff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:06.856216 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096d8ec5-9375-480d-adaf-4bc10d8757ff None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:06.873257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 129/259] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:06 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:06.880702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096d8ec5-9375-480d-adaf-4bc10d8757ff None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:06.920248 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b3e3b6-0e44-428b-bdaa-40571323f0e6 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:06.954481 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:06.956476 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:07.011029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0dd545-e718-4035-9007-91bbd3630167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:07.011603 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0dd545-e718-4035-9007-91bbd3630167 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:07.011882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0dd545-e718-4035-9007-91bbd3630167 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:07.012149 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0dd545-e718-4035-9007-91bbd3630167 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:07.043110 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0dd545-e718-4035-9007-91bbd3630167 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:07.048786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-aa0dd545-e718-4035-9007-91bbd3630167 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:47:07.048786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:07.048786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:07.048786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:07.048786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:07.048786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:07.048786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:07.048786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:07.048786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:07.048786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:07.048786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:07.048786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:07.048786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:07.048786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:07.048786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:07.048786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:07.049729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:07.049729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:07.049729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:07.049729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:07.049729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:07.049729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:07.049729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:07.049729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:07.049729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:07.049729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:07.049729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:07.049729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:07.049729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:07.049729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:07.049729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:07.049729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:07.050590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:07.050590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:07.050590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:07.050590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:07.050590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:07.050590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:07.050590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:07.050590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:07.050590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:07.050590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:07.050590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:47:07.050590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:07.050590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 131/260] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:47:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 376 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:07.057743 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0dd545-e718-4035-9007-91bbd3630167 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:07.136270 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096d8ec5-9375-480d-adaf-4bc10d8757ff None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:07.228037 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad88c06-b990-4a90-8858-834fb64262fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:07.228705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad88c06-b990-4a90-8858-834fb64262fc None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:07.228990 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad88c06-b990-4a90-8858-834fb64262fc None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:07.229269 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad88c06-b990-4a90-8858-834fb64262fc None admin] PATH_INFO: `/v3/users/eedddce36de04fcf8d98b5592b562a6f` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:07.230791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad88c06-b990-4a90-8858-834fb64262fc None admin] RBAC: Authorizing `identity:get_user(user_id=eedddce36de04fcf8d98b5592b562a6f)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:07.232254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad88c06-b990-4a90-8858-834fb64262fc None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:07.237285 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 130/261] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:47:06 2022] GET /identity/v3/users/eedddce36de04fcf8d98b5592b562a6f => generated 283 bytes in 354 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:07.247912 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad88c06-b990-4a90-8858-834fb64262fc None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:07.308347 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0dd545-e718-4035-9007-91bbd3630167 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:07.428159 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064911bd-5fd6-4272-a260-5756db8f757e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:07.428775 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064911bd-5fd6-4272-a260-5756db8f757e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:07.429050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064911bd-5fd6-4272-a260-5756db8f757e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:07.429317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064911bd-5fd6-4272-a260-5756db8f757e None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:07.429984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064911bd-5fd6-4272-a260-5756db8f757e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:07.430303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064911bd-5fd6-4272-a260-5756db8f757e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:07.431672 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064911bd-5fd6-4272-a260-5756db8f757e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:07.450780 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 132/262] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 393 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:07.457865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064911bd-5fd6-4272-a260-5756db8f757e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:07.469682 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:07.487367 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad88c06-b990-4a90-8858-834fb64262fc None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:07.552609 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d77a2d-db53-4b0f-a9d1-9fefcc6f230d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:07.553231 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d77a2d-db53-4b0f-a9d1-9fefcc6f230d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:07.553516 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d77a2d-db53-4b0f-a9d1-9fefcc6f230d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:07.553951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d77a2d-db53-4b0f-a9d1-9fefcc6f230d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:07.559237 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:07.662816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d77a2d-db53-4b0f-a9d1-9fefcc6f230d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:07.665005 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d77a2d-db53-4b0f-a9d1-9fefcc6f230d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:07.729449 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064911bd-5fd6-4272-a260-5756db8f757e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:07.794575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:07.797707 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b87dc05-031a-4634-a04b-ef69905e08ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:07.798783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b87dc05-031a-4634-a04b-ef69905e08ab None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:07.800280 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b87dc05-031a-4634-a04b-ef69905e08ab None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:07.801838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b87dc05-031a-4634-a04b-ef69905e08ab None admin] PATH_INFO: `/v3/users/eedddce36de04fcf8d98b5592b562a6f` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:07.803377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b87dc05-031a-4634-a04b-ef69905e08ab None admin] RBAC: Authorizing `identity:get_user(user_id=eedddce36de04fcf8d98b5592b562a6f)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:07.805519 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b87dc05-031a-4634-a04b-ef69905e08ab None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:07.809790 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 131/263] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 563 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:07.810469 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 133/264] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:47:07 2022] GET /identity/v3/users/eedddce36de04fcf8d98b5592b562a6f => generated 283 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:07.818397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d77a2d-db53-4b0f-a9d1-9fefcc6f230d None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:07.820049 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b87dc05-031a-4634-a04b-ef69905e08ab None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:07.986270 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d77a2d-db53-4b0f-a9d1-9fefcc6f230d None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:08.026590 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f459f350-bbe7-477a-8ab0-c001e9d17e64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:08.027444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f459f350-bbe7-477a-8ab0-c001e9d17e64 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:08.027861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f459f350-bbe7-477a-8ab0-c001e9d17e64 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:08.028275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f459f350-bbe7-477a-8ab0-c001e9d17e64 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:08.033134 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b87dc05-031a-4634-a04b-ef69905e08ab None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:08.040651 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:08.107871 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1eb3aa-c2eb-49d5-acd1-e0c1ad83cffd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:08.109986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1eb3aa-c2eb-49d5-acd1-e0c1ad83cffd None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:08.110285 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1eb3aa-c2eb-49d5-acd1-e0c1ad83cffd None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:08.110557 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1eb3aa-c2eb-49d5-acd1-e0c1ad83cffd None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:08.111857 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1eb3aa-c2eb-49d5-acd1-e0c1ad83cffd None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:08.113302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1eb3aa-c2eb-49d5-acd1-e0c1ad83cffd None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:08.116274 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 134/265] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:07 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:08.122205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1eb3aa-c2eb-49d5-acd1-e0c1ad83cffd None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:08.161541 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:08.178574 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f459f350-bbe7-477a-8ab0-c001e9d17e64 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:08.180098 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f459f350-bbe7-477a-8ab0-c001e9d17e64 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:08.266644 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1eb3aa-c2eb-49d5-acd1-e0c1ad83cffd None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:08.307978 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db544c9e-a374-41e2-9c49-32437de563d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:08.308974 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db544c9e-a374-41e2-9c49-32437de563d7 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:08.309448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db544c9e-a374-41e2-9c49-32437de563d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:08.310014 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db544c9e-a374-41e2-9c49-32437de563d7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:08.311055 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db544c9e-a374-41e2-9c49-32437de563d7 None admin] RBAC: Adding query filter params (role.id=285107fb9b2449fe9f1d29f3075a246f, scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54, user.id=eedddce36de04fcf8d98b5592b562a6f) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:08.311555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db544c9e-a374-41e2-9c49-32437de563d7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:08.318310 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db544c9e-a374-41e2-9c49-32437de563d7 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:08.329294 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 135/266] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:47:08 2022] GET /identity/v3/role_assignments?user.id=eedddce36de04fcf8d98b5592b562a6f&scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54&role.id=285107fb9b2449fe9f1d29f3075a246f => generated 608 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:08.371163 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 132/267] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 553 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:08.377486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db544c9e-a374-41e2-9c49-32437de563d7 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:08.465523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:08.561011 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db544c9e-a374-41e2-9c49-32437de563d7 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:08.603726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1036ad-57ac-46ac-8844-7718c896e3a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:08.604376 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1036ad-57ac-46ac-8844-7718c896e3a9 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:08.604657 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1036ad-57ac-46ac-8844-7718c896e3a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:08.604932 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1036ad-57ac-46ac-8844-7718c896e3a9 None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:08.606063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1036ad-57ac-46ac-8844-7718c896e3a9 None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:08.607546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1036ad-57ac-46ac-8844-7718c896e3a9 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:08.610820 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 136/268] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:08 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:08.619230 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f459f350-bbe7-477a-8ab0-c001e9d17e64 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:08.759303 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:08.761567 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c17a20-2287-4046-b8da-1e712c6e9948 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:08.761911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c17a20-2287-4046-b8da-1e712c6e9948 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:08.762231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c17a20-2287-4046-b8da-1e712c6e9948 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:08.767463 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 137/269] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:08 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:08.768127 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f459f350-bbe7-477a-8ab0-c001e9d17e64 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:08.772791 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d644d7-f754-4b9a-ae06-07bd946d599a None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:08.773083 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d644d7-f754-4b9a-ae06-07bd946d599a None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:08.773361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d644d7-f754-4b9a-ae06-07bd946d599a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:08.805112 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-11d644d7-f754-4b9a-ae06-07bd946d599a None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:08.806352 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-11d644d7-f754-4b9a-ae06-07bd946d599a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:08.840487 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e2b34d-34eb-46bf-9a1a-88083499a33d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:08.842361 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e2b34d-34eb-46bf-9a1a-88083499a33d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:08.844645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e2b34d-34eb-46bf-9a1a-88083499a33d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:08.845156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e2b34d-34eb-46bf-9a1a-88083499a33d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:08.846243 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e2b34d-34eb-46bf-9a1a-88083499a33d None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54, user.id=eedddce36de04fcf8d98b5592b562a6f) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:08.848501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e2b34d-34eb-46bf-9a1a-88083499a33d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:08.850221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e2b34d-34eb-46bf-9a1a-88083499a33d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:08.863155 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 133/270] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:47:08 2022] GET /identity/v3/role_assignments?user.id=eedddce36de04fcf8d98b5592b562a6f&scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 608 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:08.868883 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affb3e3e-059a-4ebf-a94a-c5eac706703c None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:08.869183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affb3e3e-059a-4ebf-a94a-c5eac706703c None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:08.869448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affb3e3e-059a-4ebf-a94a-c5eac706703c None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:08.872518 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 134/271] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:47:08.877957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8744911a-088a-444c-ad59-1f5d9869ef71 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:08.878327 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8744911a-088a-444c-ad59-1f5d9869ef71 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:08.878611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8744911a-088a-444c-ad59-1f5d9869ef71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:08.911199 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-8744911a-088a-444c-ad59-1f5d9869ef71 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:08.912457 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-8744911a-088a-444c-ad59-1f5d9869ef71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:08.947068 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 138/272] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:47:09.036439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 135/273] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:47:09.112040 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d875e6-228e-4623-bb9c-e9039d45ec65 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:09.112561 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d875e6-228e-4623-bb9c-e9039d45ec65 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:09.112561 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d875e6-228e-4623-bb9c-e9039d45ec65 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:09.114267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 139/274] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:09.120237 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b3c364-e9e1-4600-8bfb-2e91dc52b9c9 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:09.120531 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b3c364-e9e1-4600-8bfb-2e91dc52b9c9 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:09.120805 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b3c364-e9e1-4600-8bfb-2e91dc52b9c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:09.141410 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-f3b3c364-e9e1-4600-8bfb-2e91dc52b9c9 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:09.142771 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-f3b3c364-e9e1-4600-8bfb-2e91dc52b9c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:09.219731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 136/275] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:09.273480 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d875e6-228e-4623-bb9c-e9039d45ec65 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:09.317839 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:09.365284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b3c364-e9e1-4600-8bfb-2e91dc52b9c9 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:09.397994 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:09.411603 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d875e6-228e-4623-bb9c-e9039d45ec65 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:09.452038 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fff9c78-6bac-452e-847c-36d55df52c62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:09.453191 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff9c78-6bac-452e-847c-36d55df52c62 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:09.453738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff9c78-6bac-452e-847c-36d55df52c62 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:09.454006 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff9c78-6bac-452e-847c-36d55df52c62 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:09.455144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fff9c78-6bac-452e-847c-36d55df52c62 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:09.456575 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fff9c78-6bac-452e-847c-36d55df52c62 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:09.458347 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 140/276] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:09.459445 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20155d6e-f5ff-4ccb-8b2f-78812de63797 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:09.459726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20155d6e-f5ff-4ccb-8b2f-78812de63797 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:09.459998 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20155d6e-f5ff-4ccb-8b2f-78812de63797 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:09.461320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 141/277] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:09 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:09.464522 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20155d6e-f5ff-4ccb-8b2f-78812de63797 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:09.496888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b3c364-e9e1-4600-8bfb-2e91dc52b9c9 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:09.531175 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e7d317-4781-414c-885d-8c7494a03802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:09.531814 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e7d317-4781-414c-885d-8c7494a03802 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:09.532096 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e7d317-4781-414c-885d-8c7494a03802 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:09.532376 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e7d317-4781-414c-885d-8c7494a03802 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:09.542456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e7d317-4781-414c-885d-8c7494a03802 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:09.548152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-c3e7d317-4781-414c-885d-8c7494a03802 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:09.548152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:09.548152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:09.548152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:09.548152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:09.548152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:09.548152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:09.548152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:09.548152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:09.548152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:09.548152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:09.548152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:09.548152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:09.548152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:09.548152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:09.548152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:09.549202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:09.549202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:09.549202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:09.549202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:09.549202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:09.549202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:09.549202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:09.549202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:09.549202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:09.549202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:09.549202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:09.549202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:09.549202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:09.549202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:09.549202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:09.549202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:09.550104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:09.550104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:09.550104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:09.550104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:09.550104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:09.550104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:09.550104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:09.550104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:09.550104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:09.550104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:09.550104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:09.550104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:09.550104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 137/278] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:09.551157 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7c5aec-85fa-4ccb-bc19-c831dec9c5ad None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:09.551473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7c5aec-85fa-4ccb-bc19-c831dec9c5ad None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:09.551735 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7c5aec-85fa-4ccb-bc19-c831dec9c5ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:09.569251 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-ad7c5aec-85fa-4ccb-bc19-c831dec9c5ad None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:09.570432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-ad7c5aec-85fa-4ccb-bc19-c831dec9c5ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:09.573211 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20155d6e-f5ff-4ccb-8b2f-78812de63797 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:09.574792 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:09.609745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5cae7a-a290-4efc-bfdd-6d9d7fedf487 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:09.610383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5cae7a-a290-4efc-bfdd-6d9d7fedf487 None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:09.610661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5cae7a-a290-4efc-bfdd-6d9d7fedf487 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:09.610933 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5cae7a-a290-4efc-bfdd-6d9d7fedf487 None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:09.611582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5cae7a-a290-4efc-bfdd-6d9d7fedf487 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:09.613444 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5cae7a-a290-4efc-bfdd-6d9d7fedf487 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:09.634926 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 142/279] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:47:09 2022] POST /identity/v3/users => generated 293 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:47:09.635740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5cae7a-a290-4efc-bfdd-6d9d7fedf487 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:09.650347 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 138/280] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:09.662472 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7c5aec-85fa-4ccb-bc19-c831dec9c5ad None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:09.677555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:09.751479 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5cae7a-a290-4efc-bfdd-6d9d7fedf487 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:09.784669 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7c5aec-85fa-4ccb-bc19-c831dec9c5ad None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:09.788012 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668c4768-ac20-407d-86ab-4e53cb4533f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:09.788655 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668c4768-ac20-407d-86ab-4e53cb4533f5 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:09.788932 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668c4768-ac20-407d-86ab-4e53cb4533f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:09.789205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668c4768-ac20-407d-86ab-4e53cb4533f5 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:09.808115 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668c4768-ac20-407d-86ab-4e53cb4533f5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:09.815893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-668c4768-ac20-407d-86ab-4e53cb4533f5 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:47:09.815893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:09.815893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:09.815893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:09.815893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:09.815893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:09.815893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:09.815893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:09.815893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:09.815893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:09.815893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:09.815893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:09.815893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:09.815893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:09.815893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:09.815893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:09.816872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:09.816872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:09.816872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:09.816872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:09.816872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:09.816872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:09.816872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:09.816872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:09.816872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:09.816872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:09.816872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:09.816872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:09.816872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:09.816872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:09.816872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:09.816872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:09.817648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:09.817648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:09.817648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:09.817648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:09.817648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:09.817648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:09.817648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:09.817648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:09.817648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:09.817648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:09.817648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:47:09.817648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:09.817648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 143/281] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:09 2022] GET /identity/v3/roles/reader => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:09.823807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668c4768-ac20-407d-86ab-4e53cb4533f5 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:09.835662 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71add911-64ba-4933-bcd1-9ff890281a87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:09.836289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71add911-64ba-4933-bcd1-9ff890281a87 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:09.836569 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71add911-64ba-4933-bcd1-9ff890281a87 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:09.836848 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71add911-64ba-4933-bcd1-9ff890281a87 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:09.837533 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71add911-64ba-4933-bcd1-9ff890281a87 None admin] RBAC: Adding query filter params (name=member) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:09.837870 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71add911-64ba-4933-bcd1-9ff890281a87 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:09.840118 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71add911-64ba-4933-bcd1-9ff890281a87 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:09.848491 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:09.852697 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 139/282] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:09.860206 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71add911-64ba-4933-bcd1-9ff890281a87 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:09.959214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:09.961160 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668c4768-ac20-407d-86ab-4e53cb4533f5 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:10.003699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71add911-64ba-4933-bcd1-9ff890281a87 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:10.010056 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78a6e3b-9a88-4e33-b703-56bcc5d6852f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:10.012419 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78a6e3b-9a88-4e33-b703-56bcc5d6852f None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:10.012419 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78a6e3b-9a88-4e33-b703-56bcc5d6852f None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:10.012419 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78a6e3b-9a88-4e33-b703-56bcc5d6852f None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:10.012419 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78a6e3b-9a88-4e33-b703-56bcc5d6852f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:10.012721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78a6e3b-9a88-4e33-b703-56bcc5d6852f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:10.013846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78a6e3b-9a88-4e33-b703-56bcc5d6852f None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:10.023152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 144/283] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:09 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:10.025381 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78a6e3b-9a88-4e33-b703-56bcc5d6852f None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:10.044882 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d70a5f0-0802-4550-8f86-6f104a2189de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:10.045545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d70a5f0-0802-4550-8f86-6f104a2189de None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:10.045823 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d70a5f0-0802-4550-8f86-6f104a2189de None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:10.046681 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d70a5f0-0802-4550-8f86-6f104a2189de None admin] PATH_INFO: `/v3/users/fca2f6e0e29448b5a430980129a6a0ce` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:10.070855 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d70a5f0-0802-4550-8f86-6f104a2189de None admin] RBAC: Authorizing `identity:get_user(user_id=fca2f6e0e29448b5a430980129a6a0ce)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:10.072422 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d70a5f0-0802-4550-8f86-6f104a2189de None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:10.074789 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 140/284] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:47:09 2022] GET /identity/v3/users/fca2f6e0e29448b5a430980129a6a0ce => generated 293 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:10.075580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d70a5f0-0802-4550-8f86-6f104a2189de None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:10.129165 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:10.156504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78a6e3b-9a88-4e33-b703-56bcc5d6852f None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:10.191644 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d70a5f0-0802-4550-8f86-6f104a2189de None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:10.199543 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfc5f4e-0554-485b-bff5-d15b39a32e43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:10.200850 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfc5f4e-0554-485b-bff5-d15b39a32e43 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:10.201340 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfc5f4e-0554-485b-bff5-d15b39a32e43 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:10.201793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfc5f4e-0554-485b-bff5-d15b39a32e43 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:10.216301 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfc5f4e-0554-485b-bff5-d15b39a32e43 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:10.223736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-ddfc5f4e-0554-485b-bff5-d15b39a32e43 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:10.223736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:10.223736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:10.223736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:10.223736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:10.223736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:10.223736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:10.223736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:10.223736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:10.223736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:10.223736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:10.223736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:10.223736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:10.223736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:10.223736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:10.223736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:10.224624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:10.224624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:10.224624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:10.224624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:10.224624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:10.224624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:10.224624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:10.224624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:10.224624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:10.224624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:10.224624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:10.224624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:10.224624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:10.224624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:10.224624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:10.224624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:10.225406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:10.225406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:10.225406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:10.225406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:10.225406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:10.225406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:10.225406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:10.225406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:10.225406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:10.225406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:10.225406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:10.225406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:10.225406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 145/285] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:10.226060 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfc5f4e-0554-485b-bff5-d15b39a32e43 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:10.232318 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:10.242654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc16e568-3c60-472c-a971-1a2955be70a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:10.243638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc16e568-3c60-472c-a971-1a2955be70a0 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:10.244146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc16e568-3c60-472c-a971-1a2955be70a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:10.245074 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc16e568-3c60-472c-a971-1a2955be70a0 None admin] PATH_INFO: `/v3/users/8c293eb6aff84de19d8379d626cdde03` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:10.246505 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc16e568-3c60-472c-a971-1a2955be70a0 None admin] RBAC: Authorizing `identity:get_user(user_id=8c293eb6aff84de19d8379d626cdde03)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:10.248914 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc16e568-3c60-472c-a971-1a2955be70a0 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:10.251297 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 141/286] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:47:10 2022] GET /identity/v3/users/8c293eb6aff84de19d8379d626cdde03 => generated 297 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:10.252299 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc16e568-3c60-472c-a971-1a2955be70a0 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:10.354137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfc5f4e-0554-485b-bff5-d15b39a32e43 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:10.381228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc16e568-3c60-472c-a971-1a2955be70a0 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:10.393463 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfa0341-881b-47b0-b398-a459382fd609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:10.394280 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfa0341-881b-47b0-b398-a459382fd609 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:10.394863 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfa0341-881b-47b0-b398-a459382fd609 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:10.395213 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfa0341-881b-47b0-b398-a459382fd609 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:10.396047 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfa0341-881b-47b0-b398-a459382fd609 None admin] RBAC: Adding query filter params (role.id=34235948f8f943a99328516c2171cfff, user.id=fca2f6e0e29448b5a430980129a6a0ce, scope.system=all) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:10.396458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfa0341-881b-47b0-b398-a459382fd609 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:10.398335 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfa0341-881b-47b0-b398-a459382fd609 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:10.404448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 146/287] 10.222.0.27 () {56 vars in 1308 bytes} [Wed Sep 7 03:47:10 2022] GET /identity/v3/role_assignments?user.id=fca2f6e0e29448b5a430980129a6a0ce&scope.system=all&role.id=34235948f8f943a99328516c2171cfff => generated 226 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:10.405389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfa0341-881b-47b0-b398-a459382fd609 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:10.410070 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:10.420282 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d894b99a-1e33-438a-a4c0-e32df93f1983 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:10.421193 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d894b99a-1e33-438a-a4c0-e32df93f1983 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:10.421688 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d894b99a-1e33-438a-a4c0-e32df93f1983 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:10.422160 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d894b99a-1e33-438a-a4c0-e32df93f1983 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:10.423007 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d894b99a-1e33-438a-a4c0-e32df93f1983 None admin] RBAC: Adding query filter params (name=member) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:10.423490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d894b99a-1e33-438a-a4c0-e32df93f1983 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:10.425056 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d894b99a-1e33-438a-a4c0-e32df93f1983 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:10.434542 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 142/288] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:10.441441 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d894b99a-1e33-438a-a4c0-e32df93f1983 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:10.507359 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:10.532510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfa0341-881b-47b0-b398-a459382fd609 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:10.554076 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d894b99a-1e33-438a-a4c0-e32df93f1983 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:10.574886 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4110c420-eb73-4043-aa2f-77bace7533f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:10.575295 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4110c420-eb73-4043-aa2f-77bace7533f6 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:10.575579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4110c420-eb73-4043-aa2f-77bace7533f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:10.575852 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4110c420-eb73-4043-aa2f-77bace7533f6 None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:10.577047 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4110c420-eb73-4043-aa2f-77bace7533f6 None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:10.579141 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4110c420-eb73-4043-aa2f-77bace7533f6 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:10.586015 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 147/289] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:10 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:10.587340 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4110c420-eb73-4043-aa2f-77bace7533f6 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:10.595647 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2aa11f-7974-41c2-b545-cdded78517f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:10.596670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2aa11f-7974-41c2-b545-cdded78517f7 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:10.597189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2aa11f-7974-41c2-b545-cdded78517f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:10.597684 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2aa11f-7974-41c2-b545-cdded78517f7 None admin] PATH_INFO: `/v3/users/b44b593879b941f2a609b6a8865ed8da` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:10.599223 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2aa11f-7974-41c2-b545-cdded78517f7 None admin] RBAC: Authorizing `identity:get_user(user_id=b44b593879b941f2a609b6a8865ed8da)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:10.600993 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2aa11f-7974-41c2-b545-cdded78517f7 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:10.603270 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 143/290] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:47:10 2022] GET /identity/v3/users/b44b593879b941f2a609b6a8865ed8da => generated 297 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:10.608986 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2aa11f-7974-41c2-b545-cdded78517f7 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:10.669938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:10.709065 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4110c420-eb73-4043-aa2f-77bace7533f6 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:10.720612 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2aa11f-7974-41c2-b545-cdded78517f7 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:10.749066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0decd7d-4025-4cd8-8b2b-5cb85a2cfd3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:10.750062 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:10.751490 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0decd7d-4025-4cd8-8b2b-5cb85a2cfd3c None admin] REQUEST_METHOD: `PUT` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:10.751891 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0decd7d-4025-4cd8-8b2b-5cb85a2cfd3c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:10.752179 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0decd7d-4025-4cd8-8b2b-5cb85a2cfd3c None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54/users/8c293eb6aff84de19d8379d626cdde03/roles/e8be279b370349c8a5b9a06dd763e1b5` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:10.754598 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0decd7d-4025-4cd8-8b2b-5cb85a2cfd3c None admin] RBAC: Authorizing `identity:create_grant(project_id=72fb26d349cf4298b0ad385cd5e2ee54, user_id=8c293eb6aff84de19d8379d626cdde03, role_id=e8be279b370349c8a5b9a06dd763e1b5)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:10.756123 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0decd7d-4025-4cd8-8b2b-5cb85a2cfd3c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:10.760016 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92974c0-09b9-417f-b09c-28680b1792bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:10.760890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92974c0-09b9-417f-b09c-28680b1792bd None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:10.761483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92974c0-09b9-417f-b09c-28680b1792bd None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:10.761658 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92974c0-09b9-417f-b09c-28680b1792bd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:10.781555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 148/291] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:47:10 2022] PUT /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54/users/8c293eb6aff84de19d8379d626cdde03/roles/e8be279b370349c8a5b9a06dd763e1b5 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:47:10.841070 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92974c0-09b9-417f-b09c-28680b1792bd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:10.842606 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92974c0-09b9-417f-b09c-28680b1792bd None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:10.953223 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 144/292] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 07 03:47:10.959082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0decd7d-4025-4cd8-8b2b-5cb85a2cfd3c None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:11.119343 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0decd7d-4025-4cd8-8b2b-5cb85a2cfd3c None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:11.150917 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:11.158237 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e821bed6-21ad-4b10-90ab-d508dfd82aaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:11.158883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e821bed6-21ad-4b10-90ab-d508dfd82aaf None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:11.159164 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e821bed6-21ad-4b10-90ab-d508dfd82aaf None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:11.159437 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e821bed6-21ad-4b10-90ab-d508dfd82aaf None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:11.160586 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e821bed6-21ad-4b10-90ab-d508dfd82aaf None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:11.162019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e821bed6-21ad-4b10-90ab-d508dfd82aaf None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:11.165943 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 149/293] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:10 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:11.171654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92974c0-09b9-417f-b09c-28680b1792bd None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:11.202489 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:11.280280 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92974c0-09b9-417f-b09c-28680b1792bd None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:11.316977 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da526b8d-bee0-4976-84ff-2993e37e5068 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:11.317596 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da526b8d-bee0-4976-84ff-2993e37e5068 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:11.317877 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da526b8d-bee0-4976-84ff-2993e37e5068 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:11.318170 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da526b8d-bee0-4976-84ff-2993e37e5068 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:11.318900 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da526b8d-bee0-4976-84ff-2993e37e5068 None admin] RBAC: Adding query filter params (role.id=34235948f8f943a99328516c2171cfff, scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54, user.id=b44b593879b941f2a609b6a8865ed8da) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:11.319217 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da526b8d-bee0-4976-84ff-2993e37e5068 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:11.320655 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da526b8d-bee0-4976-84ff-2993e37e5068 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:11.326931 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 145/294] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:47:11 2022] GET /identity/v3/role_assignments?user.id=b44b593879b941f2a609b6a8865ed8da&scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54&role.id=34235948f8f943a99328516c2171cfff => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:12.183588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e542b6d-7a87-46f4-bb4c-15f3c585bf8a None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:12.184610 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e542b6d-7a87-46f4-bb4c-15f3c585bf8a None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:12.185101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e542b6d-7a87-46f4-bb4c-15f3c585bf8a None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:12.187512 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 150/295] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:12 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:12.193211 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08681d5b-39b4-4001-8ea1-0a1fbba6af07 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:12.194021 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08681d5b-39b4-4001-8ea1-0a1fbba6af07 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:12.194555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08681d5b-39b4-4001-8ea1-0a1fbba6af07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:12.214245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-08681d5b-39b4-4001-8ea1-0a1fbba6af07 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:12.215828 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-08681d5b-39b4-4001-8ea1-0a1fbba6af07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:12.306507 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 146/296] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:12.337383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dd35df-b420-4fa9-bf3d-6ad3ebb6bc0e None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:12.337687 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dd35df-b420-4fa9-bf3d-6ad3ebb6bc0e None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:12.337927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dd35df-b420-4fa9-bf3d-6ad3ebb6bc0e None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:12.339303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 151/297] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:12.344589 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5efbd19-c22a-41fa-910e-8a28f0e967dc None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:12.344879 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5efbd19-c22a-41fa-910e-8a28f0e967dc None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:12.345159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5efbd19-c22a-41fa-910e-8a28f0e967dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:12.363936 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-b5efbd19-c22a-41fa-910e-8a28f0e967dc None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:12.365119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-b5efbd19-c22a-41fa-910e-8a28f0e967dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:12.382007 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:12.436246 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 147/298] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:12.587020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44dd35df-b420-4fa9-bf3d-6ad3ebb6bc0e None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:12.693450 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44dd35df-b420-4fa9-bf3d-6ad3ebb6bc0e None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:12.719007 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5efbd19-c22a-41fa-910e-8a28f0e967dc None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:12.729108 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3400299-8e0f-4fcb-b8b9-b8a800ce48c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:12.730071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3400299-8e0f-4fcb-b8b9-b8a800ce48c7 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:12.730575 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3400299-8e0f-4fcb-b8b9-b8a800ce48c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:12.731096 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3400299-8e0f-4fcb-b8b9-b8a800ce48c7 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:12.743717 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3400299-8e0f-4fcb-b8b9-b8a800ce48c7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:12.749785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-f3400299-8e0f-4fcb-b8b9-b8a800ce48c7 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:47:12.749785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:12.749785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:12.749785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:12.749785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:12.749785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:12.749785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:12.749785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:12.749785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:12.749785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:12.749785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:12.749785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:12.749785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:12.749785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:12.749785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:12.749785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:12.750686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:12.750686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:12.750686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:12.750686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:12.750686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:12.750686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:12.750686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:12.750686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:12.750686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:12.750686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:12.750686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:12.750686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:12.750686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:12.750686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:12.750686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:12.750686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:12.751461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:12.751461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:12.751461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:12.751461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:12.751461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:12.751461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:12.751461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:12.751461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:12.751461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:12.751461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:12.751461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:47:12.751461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:12.753691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 152/299] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:12 2022] GET /identity/v3/roles/reader => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:12.759710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3400299-8e0f-4fcb-b8b9-b8a800ce48c7 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:12.823969 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:12.831139 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5efbd19-c22a-41fa-910e-8a28f0e967dc None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:12.869253 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787b3d7d-6b09-4b10-8a89-19d4ce49c342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:12.869891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787b3d7d-6b09-4b10-8a89-19d4ce49c342 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:12.870198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787b3d7d-6b09-4b10-8a89-19d4ce49c342 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:12.870459 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787b3d7d-6b09-4b10-8a89-19d4ce49c342 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:12.870934 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3400299-8e0f-4fcb-b8b9-b8a800ce48c7 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:12.871572 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787b3d7d-6b09-4b10-8a89-19d4ce49c342 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:12.873116 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787b3d7d-6b09-4b10-8a89-19d4ce49c342 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:12.880039 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 148/300] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:12.880893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787b3d7d-6b09-4b10-8a89-19d4ce49c342 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:12.904961 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:12.911733 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d60ff6c-6b47-4582-a456-4e4165dd247b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:12.912749 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d60ff6c-6b47-4582-a456-4e4165dd247b None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:12.913228 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d60ff6c-6b47-4582-a456-4e4165dd247b None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:12.913698 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d60ff6c-6b47-4582-a456-4e4165dd247b None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:12.914714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d60ff6c-6b47-4582-a456-4e4165dd247b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:12.915236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d60ff6c-6b47-4582-a456-4e4165dd247b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:12.916883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d60ff6c-6b47-4582-a456-4e4165dd247b None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:12.930261 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 153/301] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:12 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:12.933445 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d60ff6c-6b47-4582-a456-4e4165dd247b None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:12.995146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787b3d7d-6b09-4b10-8a89-19d4ce49c342 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:13.033026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd74652-85a4-44c4-b1c7-15c07f73eed9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:13.034046 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd74652-85a4-44c4-b1c7-15c07f73eed9 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:13.034046 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd74652-85a4-44c4-b1c7-15c07f73eed9 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:13.034362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd74652-85a4-44c4-b1c7-15c07f73eed9 None admin] PATH_INFO: `/v3/groups` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:13.034933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd74652-85a4-44c4-b1c7-15c07f73eed9 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:13.036374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd74652-85a4-44c4-b1c7-15c07f73eed9 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:13.048049 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d60ff6c-6b47-4582-a456-4e4165dd247b None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:13.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 149/302] 10.222.0.27 () {60 vars in 1134 bytes} [Wed Sep 7 03:47:12 2022] POST /identity/v3/groups => generated 229 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:47:13.063145 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abbfa84-a6d4-4f8f-afaa-36aae0a03f6d None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:13.063516 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abbfa84-a6d4-4f8f-afaa-36aae0a03f6d None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:13.063805 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abbfa84-a6d4-4f8f-afaa-36aae0a03f6d None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:13.065519 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 150/303] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:47:13.071021 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8abf1fa-cbdd-4954-98e7-dad3f78716ab None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:13.071470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8abf1fa-cbdd-4954-98e7-dad3f78716ab None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:13.071839 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8abf1fa-cbdd-4954-98e7-dad3f78716ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:13.080898 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:13.094063 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-f8abf1fa-cbdd-4954-98e7-dad3f78716ab None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:13.095479 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-f8abf1fa-cbdd-4954-98e7-dad3f78716ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:13.095742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560e7005-0f67-4a15-905d-9c04beb5c9c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:13.096598 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560e7005-0f67-4a15-905d-9c04beb5c9c9 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:13.097022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560e7005-0f67-4a15-905d-9c04beb5c9c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:13.097455 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560e7005-0f67-4a15-905d-9c04beb5c9c9 None admin] PATH_INFO: `/v3/users/ea0f0e12b27343b4808a542e80985a93` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:13.114356 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560e7005-0f67-4a15-905d-9c04beb5c9c9 None admin] RBAC: Authorizing `identity:get_user(user_id=ea0f0e12b27343b4808a542e80985a93)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:13.116205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560e7005-0f67-4a15-905d-9c04beb5c9c9 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:13.123550 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 154/304] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:47:12 2022] GET /identity/v3/users/ea0f0e12b27343b4808a542e80985a93 => generated 293 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:13.126102 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560e7005-0f67-4a15-905d-9c04beb5c9c9 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:13.176786 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 151/305] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:13.241690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560e7005-0f67-4a15-905d-9c04beb5c9c9 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:13.280048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5976c511-b621-43db-95a1-031443b6b9b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:13.281218 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976c511-b621-43db-95a1-031443b6b9b7 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:13.281496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976c511-b621-43db-95a1-031443b6b9b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:13.281764 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976c511-b621-43db-95a1-031443b6b9b7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:13.282560 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5976c511-b621-43db-95a1-031443b6b9b7 None admin] RBAC: Adding query filter params (role.id=e8be279b370349c8a5b9a06dd763e1b5, user.id=ea0f0e12b27343b4808a542e80985a93, scope.system=all) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:13.282870 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5976c511-b621-43db-95a1-031443b6b9b7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:13.284357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5976c511-b621-43db-95a1-031443b6b9b7 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:13.290315 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 155/306] 10.222.0.27 () {56 vars in 1308 bytes} [Wed Sep 7 03:47:13 2022] GET /identity/v3/role_assignments?user.id=ea0f0e12b27343b4808a542e80985a93&scope.system=all&role.id=e8be279b370349c8a5b9a06dd763e1b5 => generated 226 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:13.414391 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf4ab00-e472-4713-9acd-7d07b46b3dc3 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:13.414900 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf4ab00-e472-4713-9acd-7d07b46b3dc3 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:13.414972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf4ab00-e472-4713-9acd-7d07b46b3dc3 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:13.416713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 152/307] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:13.428158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0276def-0c7c-4062-aa85-bd667e7333b9 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:13.428590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0276def-0c7c-4062-aa85-bd667e7333b9 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:13.428965 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0276def-0c7c-4062-aa85-bd667e7333b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:13.450094 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-f0276def-0c7c-4062-aa85-bd667e7333b9 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:13.451331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-f0276def-0c7c-4062-aa85-bd667e7333b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:13.478482 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf4ab00-e472-4713-9acd-7d07b46b3dc3 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:13.497762 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:13.501049 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:13.538175 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 156/308] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:13.613074 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf4ab00-e472-4713-9acd-7d07b46b3dc3 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:13.651186 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7f4c45-6134-4777-be39-ac3cebccad42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:13.652182 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7f4c45-6134-4777-be39-ac3cebccad42 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:13.652691 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7f4c45-6134-4777-be39-ac3cebccad42 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:13.653164 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7f4c45-6134-4777-be39-ac3cebccad42 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:13.665046 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7f4c45-6134-4777-be39-ac3cebccad42 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:13.670471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-2f7f4c45-6134-4777-be39-ac3cebccad42 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:13.670471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:13.670471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:13.670471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:13.670471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:13.670471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:13.670471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:13.670471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:13.670471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:13.670471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:13.670471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:13.670471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:13.670471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:13.670471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:13.670471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:13.670471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:13.671635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:13.671635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:13.671635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:13.671635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:13.671635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:13.671635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:13.671635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:13.671635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:13.671635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:13.671635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:13.671635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:13.671635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:13.671635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:13.671635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:13.671635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:13.671635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:13.672430 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:13.672430 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:13.672430 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:13.672430 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:13.672430 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:13.672430 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:13.672430 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:13.672430 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:13.672430 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:13.672430 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:13.672430 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:13.672430 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:13.675007 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 153/309] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:13.680967 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0276def-0c7c-4062-aa85-bd667e7333b9 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:13.788988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0276def-0c7c-4062-aa85-bd667e7333b9 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:13.806210 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7f4c45-6134-4777-be39-ac3cebccad42 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:13.832793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471f536b-242c-4532-8340-855d7d0af229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:13.833435 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471f536b-242c-4532-8340-855d7d0af229 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:13.833687 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471f536b-242c-4532-8340-855d7d0af229 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:13.833956 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471f536b-242c-4532-8340-855d7d0af229 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:13.834661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471f536b-242c-4532-8340-855d7d0af229 None admin] RBAC: Adding query filter params (name=member) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:13.834959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471f536b-242c-4532-8340-855d7d0af229 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:13.836409 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471f536b-242c-4532-8340-855d7d0af229 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:13.845620 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 157/310] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:13.851674 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471f536b-242c-4532-8340-855d7d0af229 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:13.887458 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:13.895446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:13.932512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7f4c45-6134-4777-be39-ac3cebccad42 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:13.974058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2a3bb0-3c10-46b3-b910-8d588c124153 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:13.974975 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2a3bb0-3c10-46b3-b910-8d588c124153 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:13.975359 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2a3bb0-3c10-46b3-b910-8d588c124153 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:13.975697 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2a3bb0-3c10-46b3-b910-8d588c124153 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:13.985017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471f536b-242c-4532-8340-855d7d0af229 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:13.987484 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2a3bb0-3c10-46b3-b910-8d588c124153 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:13.992649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-cf2a3bb0-3c10-46b3-b910-8d588c124153 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:47:13.992649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:13.992649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:13.992649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:13.992649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:13.992649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:13.992649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:13.992649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:13.992649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:13.992649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:13.992649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:13.992649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:13.992649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:13.992649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:13.992649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:13.992649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:13.993504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:13.993504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:13.993504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:13.993504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:13.993504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:13.993504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:13.993504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:13.993504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:13.993504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:13.993504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:13.993504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:13.993504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:13.993504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:13.993504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:13.993504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:13.993504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:13.994312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:13.994312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:13.994312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:13.994312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:13.994312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:13.994312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:13.994312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:13.994312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:13.994312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:13.994312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:13.994312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:47:13.994312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:13.994312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 154/311] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:13 2022] GET /identity/v3/roles/reader => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:14.006241 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2a3bb0-3c10-46b3-b910-8d588c124153 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:14.029773 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39507c1-64c0-4e7c-819f-6c1d5f211b74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:14.030740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39507c1-64c0-4e7c-819f-6c1d5f211b74 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:14.031073 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39507c1-64c0-4e7c-819f-6c1d5f211b74 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:14.031299 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39507c1-64c0-4e7c-819f-6c1d5f211b74 None admin] PATH_INFO: `/v3/users/fca2f6e0e29448b5a430980129a6a0ce` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:14.032523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39507c1-64c0-4e7c-819f-6c1d5f211b74 None admin] RBAC: Authorizing `identity:get_user(user_id=fca2f6e0e29448b5a430980129a6a0ce)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:14.033954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39507c1-64c0-4e7c-819f-6c1d5f211b74 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:14.040340 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 158/312] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:47:13 2022] GET /identity/v3/users/fca2f6e0e29448b5a430980129a6a0ce => generated 293 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:14.041224 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39507c1-64c0-4e7c-819f-6c1d5f211b74 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:14.130907 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2a3bb0-3c10-46b3-b910-8d588c124153 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:14.160595 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:14.168039 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e1785e-ab6f-48fb-8476-b8155d11f6b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:14.168566 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39507c1-64c0-4e7c-819f-6c1d5f211b74 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:14.168674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e1785e-ab6f-48fb-8476-b8155d11f6b4 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:14.168913 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e1785e-ab6f-48fb-8476-b8155d11f6b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:14.169183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e1785e-ab6f-48fb-8476-b8155d11f6b4 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:14.169845 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e1785e-ab6f-48fb-8476-b8155d11f6b4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:14.170167 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e1785e-ab6f-48fb-8476-b8155d11f6b4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:14.171790 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e1785e-ab6f-48fb-8476-b8155d11f6b4 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:14.172562 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:14.181421 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 155/313] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:13 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:14.188153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e1785e-ab6f-48fb-8476-b8155d11f6b4 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:14.215387 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e23d02-7ce2-4ac6-8dfd-f8c17a38740e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:14.216470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e23d02-7ce2-4ac6-8dfd-f8c17a38740e None admin] REQUEST_METHOD: `PUT` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:14.217014 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e23d02-7ce2-4ac6-8dfd-f8c17a38740e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:14.217482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e23d02-7ce2-4ac6-8dfd-f8c17a38740e None admin] PATH_INFO: `/v3/system/users/fca2f6e0e29448b5a430980129a6a0ce/roles/34235948f8f943a99328516c2171cfff` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:14.219946 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e23d02-7ce2-4ac6-8dfd-f8c17a38740e None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=fca2f6e0e29448b5a430980129a6a0ce, role_id=34235948f8f943a99328516c2171cfff)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:14.221610 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e23d02-7ce2-4ac6-8dfd-f8c17a38740e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:14.240027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 159/314] 10.222.0.27 () {58 vars in 1334 bytes} [Wed Sep 7 03:47:14 2022] PUT /identity/v3/system/users/fca2f6e0e29448b5a430980129a6a0ce/roles/34235948f8f943a99328516c2171cfff => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:47:14.318719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e1785e-ab6f-48fb-8476-b8155d11f6b4 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:14.362290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64acc82-13e5-470b-a29e-a54f5c218f44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:14.363303 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64acc82-13e5-470b-a29e-a54f5c218f44 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:14.363815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64acc82-13e5-470b-a29e-a54f5c218f44 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:14.364274 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64acc82-13e5-470b-a29e-a54f5c218f44 None admin] PATH_INFO: `/v3/users/8c293eb6aff84de19d8379d626cdde03` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:14.365666 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64acc82-13e5-470b-a29e-a54f5c218f44 None admin] RBAC: Authorizing `identity:get_user(user_id=8c293eb6aff84de19d8379d626cdde03)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:14.367461 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64acc82-13e5-470b-a29e-a54f5c218f44 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:14.369666 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 156/315] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:47:14 2022] GET /identity/v3/users/8c293eb6aff84de19d8379d626cdde03 => generated 297 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:14.375305 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e23d02-7ce2-4ac6-8dfd-f8c17a38740e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:14.489033 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e23d02-7ce2-4ac6-8dfd-f8c17a38740e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:14.524931 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ccc0a3-6cbd-4d49-b9bb-419eed1c23c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:14.525590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ccc0a3-6cbd-4d49-b9bb-419eed1c23c0 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:14.525866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ccc0a3-6cbd-4d49-b9bb-419eed1c23c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:14.526206 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ccc0a3-6cbd-4d49-b9bb-419eed1c23c0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:14.565205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:14.603558 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ccc0a3-6cbd-4d49-b9bb-419eed1c23c0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:14.605291 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ccc0a3-6cbd-4d49-b9bb-419eed1c23c0 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:14.709309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 160/316] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:14.715184 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64acc82-13e5-470b-a29e-a54f5c218f44 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:14.778397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:14.821544 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64acc82-13e5-470b-a29e-a54f5c218f44 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:14.856897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ead5d3-9e9f-4b5e-90af-6386d08a0f18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:14.857459 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ead5d3-9e9f-4b5e-90af-6386d08a0f18 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:14.857741 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ead5d3-9e9f-4b5e-90af-6386d08a0f18 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:14.858014 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ead5d3-9e9f-4b5e-90af-6386d08a0f18 None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:14.859397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ead5d3-9e9f-4b5e-90af-6386d08a0f18 None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:14.861246 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ead5d3-9e9f-4b5e-90af-6386d08a0f18 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:14.863228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 157/317] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:14 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:14.869132 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ccc0a3-6cbd-4d49-b9bb-419eed1c23c0 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:14.940227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:14.976049 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ccc0a3-6cbd-4d49-b9bb-419eed1c23c0 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:15.012051 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c14c86-94e9-4e3a-8bf2-116ce65c17e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:15.012724 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c14c86-94e9-4e3a-8bf2-116ce65c17e2 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:15.013002 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c14c86-94e9-4e3a-8bf2-116ce65c17e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:15.013272 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c14c86-94e9-4e3a-8bf2-116ce65c17e2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:15.014039 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c14c86-94e9-4e3a-8bf2-116ce65c17e2 None admin] RBAC: Adding query filter params (role.id=e8be279b370349c8a5b9a06dd763e1b5, scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54, user.id=8c293eb6aff84de19d8379d626cdde03) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:15.014382 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c14c86-94e9-4e3a-8bf2-116ce65c17e2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:15.015908 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c14c86-94e9-4e3a-8bf2-116ce65c17e2 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:15.022573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 161/318] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:47:14 2022] GET /identity/v3/role_assignments?user.id=8c293eb6aff84de19d8379d626cdde03&scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54&role.id=e8be279b370349c8a5b9a06dd763e1b5 => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:15.365979 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8ff9ed-554d-4e12-a64f-847a96d7de3e None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:15.366345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8ff9ed-554d-4e12-a64f-847a96d7de3e None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:15.366575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8ff9ed-554d-4e12-a64f-847a96d7de3e None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:15.368090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 158/319] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:15.373203 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dac58a4-c550-4c38-8e75-f3d043eb8cf0 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:15.373554 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dac58a4-c550-4c38-8e75-f3d043eb8cf0 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:15.373925 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dac58a4-c550-4c38-8e75-f3d043eb8cf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:15.394241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-6dac58a4-c550-4c38-8e75-f3d043eb8cf0 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:15.395483 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-6dac58a4-c550-4c38-8e75-f3d043eb8cf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:15.469610 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 162/320] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:15.508571 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5cde0a-9aca-4097-b0e6-59c9280add02 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:15.508857 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5cde0a-9aca-4097-b0e6-59c9280add02 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:15.509181 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5cde0a-9aca-4097-b0e6-59c9280add02 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:15.510638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 159/321] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:15.516285 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d208b24-dff2-4b6f-9e82-d839e4344ec6 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:15.516573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d208b24-dff2-4b6f-9e82-d839e4344ec6 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:15.516843 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d208b24-dff2-4b6f-9e82-d839e4344ec6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:15.534768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-6d208b24-dff2-4b6f-9e82-d839e4344ec6 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:15.535957 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-6d208b24-dff2-4b6f-9e82-d839e4344ec6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:15.594626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:15.605965 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 163/322] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:15.736874 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5cde0a-9aca-4097-b0e6-59c9280add02 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:15.839580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5cde0a-9aca-4097-b0e6-59c9280add02 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:15.865827 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:15.873247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d208b24-dff2-4b6f-9e82-d839e4344ec6 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:15.875891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310aef8e-5f77-4b20-8ccf-2c9e06ce0150 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:15.876533 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310aef8e-5f77-4b20-8ccf-2c9e06ce0150 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:15.876808 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310aef8e-5f77-4b20-8ccf-2c9e06ce0150 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:15.877078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310aef8e-5f77-4b20-8ccf-2c9e06ce0150 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:15.878211 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310aef8e-5f77-4b20-8ccf-2c9e06ce0150 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:15.879649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310aef8e-5f77-4b20-8ccf-2c9e06ce0150 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:15.881354 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 160/323] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:15.886549 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310aef8e-5f77-4b20-8ccf-2c9e06ce0150 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:15.974491 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d208b24-dff2-4b6f-9e82-d839e4344ec6 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:15.993103 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310aef8e-5f77-4b20-8ccf-2c9e06ce0150 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:16.013757 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a6c385-3d06-4ed3-8afc-542aa40147b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:16.014428 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a6c385-3d06-4ed3-8afc-542aa40147b5 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:16.015345 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a6c385-3d06-4ed3-8afc-542aa40147b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:16.015739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a6c385-3d06-4ed3-8afc-542aa40147b5 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:16.026365 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a6c385-3d06-4ed3-8afc-542aa40147b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:16.030865 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e987c5-e18c-472b-b184-b2b931fac786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:16.031577 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e987c5-e18c-472b-b184-b2b931fac786 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:16.031738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-c3a6c385-3d06-4ed3-8afc-542aa40147b5 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:47:16.031738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:16.031738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:16.031738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:16.031738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:16.031738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:16.031738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:16.031738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:16.031738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:16.031738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:16.031738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:16.031738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:16.031738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:16.031738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:16.031738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:16.031738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:16.032571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:16.032571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:16.032571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:16.032571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:16.032571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:16.032571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:16.032571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:16.032571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:16.032571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:16.032571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:16.032571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:16.032571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:16.032571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:16.032571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:16.032571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:16.032571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:16.033372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:16.033372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:16.033372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:16.033372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:16.033372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:16.033372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:16.033372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:16.033372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:16.033372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:16.033372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:16.033372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:47:16.033372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:16.033957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e987c5-e18c-472b-b184-b2b931fac786 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:16.033957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e987c5-e18c-472b-b184-b2b931fac786 None admin] PATH_INFO: `/v3/groups` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:16.033957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e987c5-e18c-472b-b184-b2b931fac786 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:16.034141 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 164/324] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:15 2022] GET /identity/v3/roles/reader => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:16.034209 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e987c5-e18c-472b-b184-b2b931fac786 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:16.040788 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a6c385-3d06-4ed3-8afc-542aa40147b5 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:16.051860 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 161/325] 10.222.0.27 () {60 vars in 1134 bytes} [Wed Sep 7 03:47:15 2022] POST /identity/v3/groups => generated 226 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 07 03:47:16.103514 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:16.156680 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a6c385-3d06-4ed3-8afc-542aa40147b5 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:16.194950 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500eeea0-6a33-477c-be54-4d8e1021adea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:16.195575 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500eeea0-6a33-477c-be54-4d8e1021adea None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:16.195853 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500eeea0-6a33-477c-be54-4d8e1021adea None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:16.196127 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500eeea0-6a33-477c-be54-4d8e1021adea None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:16.196808 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500eeea0-6a33-477c-be54-4d8e1021adea None admin] RBAC: Adding query filter params (name=reader) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:16.197124 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500eeea0-6a33-477c-be54-4d8e1021adea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:16.198562 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500eeea0-6a33-477c-be54-4d8e1021adea None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:16.208534 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 165/326] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:16 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:16.214280 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e987c5-e18c-472b-b184-b2b931fac786 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:16.275778 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:16.322224 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e987c5-e18c-472b-b184-b2b931fac786 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:16.360726 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de92b624-fb41-452f-b16d-6dc61754306d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:16.361306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de92b624-fb41-452f-b16d-6dc61754306d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:16.361587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de92b624-fb41-452f-b16d-6dc61754306d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:16.361861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de92b624-fb41-452f-b16d-6dc61754306d None admin] PATH_INFO: `/v3/users/ea0f0e12b27343b4808a542e80985a93` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:16.363041 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de92b624-fb41-452f-b16d-6dc61754306d None admin] RBAC: Authorizing `identity:get_user(user_id=ea0f0e12b27343b4808a542e80985a93)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:16.364514 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de92b624-fb41-452f-b16d-6dc61754306d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:16.366289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 162/327] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:47:16 2022] GET /identity/v3/users/ea0f0e12b27343b4808a542e80985a93 => generated 293 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:16.372295 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500eeea0-6a33-477c-be54-4d8e1021adea None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:16.495655 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500eeea0-6a33-477c-be54-4d8e1021adea None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:16.516449 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:16.553718 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f31f5f8-cb21-46d1-aecc-208345998c4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:16.554772 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f31f5f8-cb21-46d1-aecc-208345998c4a None admin] REQUEST_METHOD: `PUT` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:16.555324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f31f5f8-cb21-46d1-aecc-208345998c4a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:16.555614 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f31f5f8-cb21-46d1-aecc-208345998c4a None admin] PATH_INFO: `/v3/system/users/ea0f0e12b27343b4808a542e80985a93/roles/e8be279b370349c8a5b9a06dd763e1b5` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:16.557171 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f31f5f8-cb21-46d1-aecc-208345998c4a None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=ea0f0e12b27343b4808a542e80985a93, role_id=e8be279b370349c8a5b9a06dd763e1b5)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:16.558601 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f31f5f8-cb21-46d1-aecc-208345998c4a None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:16.559125 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3892c3-fc4c-47e8-9f36-3904a2eb6e53 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:16.559694 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3892c3-fc4c-47e8-9f36-3904a2eb6e53 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:16.560416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3892c3-fc4c-47e8-9f36-3904a2eb6e53 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:16.565849 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 163/328] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:16 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:16.569706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 166/329] 10.222.0.27 () {58 vars in 1334 bytes} [Wed Sep 7 03:47:16 2022] PUT /identity/v3/system/users/ea0f0e12b27343b4808a542e80985a93/roles/e8be279b370349c8a5b9a06dd763e1b5 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:47:16.571721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117e2450-e322-4dc5-b35f-de04652435cf None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:16.572012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117e2450-e322-4dc5-b35f-de04652435cf None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:16.572324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117e2450-e322-4dc5-b35f-de04652435cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:16.606007 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-117e2450-e322-4dc5-b35f-de04652435cf None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:16.607427 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-117e2450-e322-4dc5-b35f-de04652435cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:16.693637 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 164/330] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:16.972651 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f31f5f8-cb21-46d1-aecc-208345998c4a None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:17.101127 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f31f5f8-cb21-46d1-aecc-208345998c4a None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:17.145842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327c219a-8795-41d6-9821-c4b067f7d53b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:17.146531 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327c219a-8795-41d6-9821-c4b067f7d53b None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:17.146811 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327c219a-8795-41d6-9821-c4b067f7d53b None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:17.147080 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327c219a-8795-41d6-9821-c4b067f7d53b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:17.161618 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327c219a-8795-41d6-9821-c4b067f7d53b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:17.167842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-327c219a-8795-41d6-9821-c4b067f7d53b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:17.167842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:17.167842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:17.167842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:17.167842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:17.167842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:17.167842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:17.167842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:17.167842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:17.167842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:17.167842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:17.167842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:17.167842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:17.167842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:17.167842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:17.167842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:17.168715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:17.168715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:17.168715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:17.168715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:17.168715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:17.168715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:17.168715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:17.168715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:17.168715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:17.168715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:17.168715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:17.168715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:17.168715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:17.168715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:17.168715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:17.168715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:17.169501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:17.169501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:17.169501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:17.169501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:17.169501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:17.169501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:17.169501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:17.169501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:17.169501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:17.169501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:17.169501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:17.169501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:17.169501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 167/331] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:16 2022] GET /identity/v3/roles/member => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:17.176630 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117e2450-e322-4dc5-b35f-de04652435cf None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:17.247729 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:17.310678 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117e2450-e322-4dc5-b35f-de04652435cf None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:17.369362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cb9ff9-cfbb-4d4c-8e74-f4d2b20f8b22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:17.369994 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cb9ff9-cfbb-4d4c-8e74-f4d2b20f8b22 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:17.370303 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cb9ff9-cfbb-4d4c-8e74-f4d2b20f8b22 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:17.370642 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cb9ff9-cfbb-4d4c-8e74-f4d2b20f8b22 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:17.371319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cb9ff9-cfbb-4d4c-8e74-f4d2b20f8b22 None admin] RBAC: Adding query filter params (name=member) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:17.371669 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cb9ff9-cfbb-4d4c-8e74-f4d2b20f8b22 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:17.373062 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cb9ff9-cfbb-4d4c-8e74-f4d2b20f8b22 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:17.385221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 165/332] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:17 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:17.390718 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327c219a-8795-41d6-9821-c4b067f7d53b None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:17.464233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:17.544125 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327c219a-8795-41d6-9821-c4b067f7d53b None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:17.597415 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e60aee4-55d5-4380-9961-962883f409e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:17.598430 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e60aee4-55d5-4380-9961-962883f409e4 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:17.598905 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e60aee4-55d5-4380-9961-962883f409e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:17.599370 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e60aee4-55d5-4380-9961-962883f409e4 None admin] PATH_INFO: `/v3/users/fca2f6e0e29448b5a430980129a6a0ce` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:17.606786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e60aee4-55d5-4380-9961-962883f409e4 None admin] RBAC: Authorizing `identity:get_user(user_id=fca2f6e0e29448b5a430980129a6a0ce)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:17.608974 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e60aee4-55d5-4380-9961-962883f409e4 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:17.611487 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 168/333] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:47:17 2022] GET /identity/v3/users/fca2f6e0e29448b5a430980129a6a0ce => generated 293 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:17.617073 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cb9ff9-cfbb-4d4c-8e74-f4d2b20f8b22 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:17.754791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cb9ff9-cfbb-4d4c-8e74-f4d2b20f8b22 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:17.781799 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:17.802840 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4266b42-b520-42bd-a01c-aba21eae9130 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:17.803379 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4266b42-b520-42bd-a01c-aba21eae9130 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:17.803717 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4266b42-b520-42bd-a01c-aba21eae9130 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:17.803993 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4266b42-b520-42bd-a01c-aba21eae9130 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:17.804730 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4266b42-b520-42bd-a01c-aba21eae9130 None admin] RBAC: Adding query filter params (role.id=34235948f8f943a99328516c2171cfff, user.id=fca2f6e0e29448b5a430980129a6a0ce, scope.system=all) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:17.805039 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4266b42-b520-42bd-a01c-aba21eae9130 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:17.806507 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4266b42-b520-42bd-a01c-aba21eae9130 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:17.824557 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 166/334] 10.222.0.27 () {56 vars in 1308 bytes} [Wed Sep 7 03:47:17 2022] GET /identity/v3/role_assignments?user.id=fca2f6e0e29448b5a430980129a6a0ce&scope.system=all&role.id=34235948f8f943a99328516c2171cfff => generated 510 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:19.191817 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feb591b-0dd6-4254-9486-0a7e3b02cbd5 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:19.192887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feb591b-0dd6-4254-9486-0a7e3b02cbd5 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:19.193370 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feb591b-0dd6-4254-9486-0a7e3b02cbd5 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:19.195031 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 169/335] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:19.200997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a80711-1c10-47c5-bcd1-e0a7b5afa6b1 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:19.201912 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a80711-1c10-47c5-bcd1-e0a7b5afa6b1 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:19.203118 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a80711-1c10-47c5-bcd1-e0a7b5afa6b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:19.224900 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-38a80711-1c10-47c5-bcd1-e0a7b5afa6b1 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:19.232011 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-38a80711-1c10-47c5-bcd1-e0a7b5afa6b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:19.307531 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1611f9f1-5691-432f-b92f-7324e6ac94b1 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:19.307874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1611f9f1-5691-432f-b92f-7324e6ac94b1 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:19.308090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1611f9f1-5691-432f-b92f-7324e6ac94b1 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:19.314232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 170/336] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:19.322134 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 167/337] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:19.323726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e287798-46ba-4758-9246-494d817d3aa2 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:19.324020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e287798-46ba-4758-9246-494d817d3aa2 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:19.324301 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e287798-46ba-4758-9246-494d817d3aa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:19.355152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-0e287798-46ba-4758-9246-494d817d3aa2 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:19.356388 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-0e287798-46ba-4758-9246-494d817d3aa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:19.384183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cd6601-c6c6-46ba-9efa-d3290f7e5117 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:19.384482 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cd6601-c6c6-46ba-9efa-d3290f7e5117 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:19.384727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cd6601-c6c6-46ba-9efa-d3290f7e5117 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:19.386513 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 168/338] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:19.393491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0f3cb6-15e7-4e32-ae9a-49176667f17a None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:19.393779 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0f3cb6-15e7-4e32-ae9a-49176667f17a None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:19.394038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0f3cb6-15e7-4e32-ae9a-49176667f17a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:19.424121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-df0f3cb6-15e7-4e32-ae9a-49176667f17a None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:19.425313 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-df0f3cb6-15e7-4e32-ae9a-49176667f17a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:19.489699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 171/339] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:19.542825 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 169/340] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:47:19.632346 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e287798-46ba-4758-9246-494d817d3aa2 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:19.687799 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:19.770654 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e287798-46ba-4758-9246-494d817d3aa2 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:19.821449 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07f866b-ab50-45a0-ad9c-b4afb81844c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:19.822479 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07f866b-ab50-45a0-ad9c-b4afb81844c4 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:19.822992 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07f866b-ab50-45a0-ad9c-b4afb81844c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:19.823485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07f866b-ab50-45a0-ad9c-b4afb81844c4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:19.839468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07f866b-ab50-45a0-ad9c-b4afb81844c4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:19.844784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-d07f866b-ab50-45a0-ad9c-b4afb81844c4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:19.844784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:19.844784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:19.844784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:19.844784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:19.844784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:19.844784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:19.844784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:19.844784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:19.844784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:19.844784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:19.844784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:19.844784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:19.844784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:19.844784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:19.844784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:19.845673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:19.845673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:19.845673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:19.845673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:19.845673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:19.845673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:19.845673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:19.845673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:19.845673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:19.845673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:19.845673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:19.845673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:19.845673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:19.845673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:19.845673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:19.845673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:19.846486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:19.846486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:19.846486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:19.846486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:19.846486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:19.846486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:19.846486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:19.846486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:19.846486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:19.846486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:19.846486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:19.846486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:19.849065 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 172/341] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:19.855238 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0f3cb6-15e7-4e32-ae9a-49176667f17a None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:19.866705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07f866b-ab50-45a0-ad9c-b4afb81844c4 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:19.925766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:20.056904 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:20.067555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07f866b-ab50-45a0-ad9c-b4afb81844c4 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:20.085919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0f3cb6-15e7-4e32-ae9a-49176667f17a None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:20.107494 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a32f5d-86ad-4bae-8504-a6cb25f502e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:20.108251 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a32f5d-86ad-4bae-8504-a6cb25f502e8 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:20.108705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a32f5d-86ad-4bae-8504-a6cb25f502e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:20.109589 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a32f5d-86ad-4bae-8504-a6cb25f502e8 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:20.122237 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a32f5d-86ad-4bae-8504-a6cb25f502e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:20.129569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-d3a32f5d-86ad-4bae-8504-a6cb25f502e8 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:47:20.129569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:20.129569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:20.129569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:20.129569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:20.129569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:20.129569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:20.129569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:20.129569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:20.129569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:20.129569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:20.129569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:20.129569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:20.129569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:20.129569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:20.129569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:20.130542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:20.130542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:20.130542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:20.130542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:20.130542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:20.130542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:20.130542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:20.130542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:20.130542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:20.130542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:20.130542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:20.130542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:20.130542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:20.130542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:20.130542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:20.130542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:20.131322 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:20.131322 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:20.131322 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:20.131322 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:20.131322 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:20.131322 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:20.131322 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:20.131322 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:20.131322 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:20.131322 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:20.131322 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 07 03:47:20.131322 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:20.134238 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 173/342] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:19 2022] GET /identity/v3/roles/reader => generated 84 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:20.135850 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a32f5d-86ad-4bae-8504-a6cb25f502e8 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:20.143578 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce084a8e-9bfe-4982-8472-d52c32d86fc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:20.144610 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce084a8e-9bfe-4982-8472-d52c32d86fc6 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:20.145091 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce084a8e-9bfe-4982-8472-d52c32d86fc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:20.145559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce084a8e-9bfe-4982-8472-d52c32d86fc6 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:20.146500 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce084a8e-9bfe-4982-8472-d52c32d86fc6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:20.147013 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce084a8e-9bfe-4982-8472-d52c32d86fc6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:20.148656 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce084a8e-9bfe-4982-8472-d52c32d86fc6 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:20.164423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 170/343] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:20.166019 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc48a44-569d-4e56-b35d-6bee95581cfc None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:20.166679 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc48a44-569d-4e56-b35d-6bee95581cfc None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:20.170206 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc48a44-569d-4e56-b35d-6bee95581cfc None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:20.172088 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 171/344] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:20 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:20.174061 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc48a44-569d-4e56-b35d-6bee95581cfc None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:20.282811 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a32f5d-86ad-4bae-8504-a6cb25f502e8 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:20.305797 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:20.314938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc48a44-569d-4e56-b35d-6bee95581cfc None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:20.334840 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd264020-c73c-40de-a775-42f12f61e8d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:20.335793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd264020-c73c-40de-a775-42f12f61e8d7 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:20.336253 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd264020-c73c-40de-a775-42f12f61e8d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:20.336748 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd264020-c73c-40de-a775-42f12f61e8d7 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:20.349547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd264020-c73c-40de-a775-42f12f61e8d7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:20.354887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-bd264020-c73c-40de-a775-42f12f61e8d7 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:20.354887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:20.354887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:20.354887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:20.354887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:20.354887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:20.354887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:20.354887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:20.354887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:20.354887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:20.354887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:20.354887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:20.354887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:20.354887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:20.354887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:20.354887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:20.355755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:20.355755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:20.355755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:20.355755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:20.355755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:20.355755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:20.355755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:20.355755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:20.355755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:20.355755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:20.355755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:20.355755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:20.355755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:20.355755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:20.355755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:20.355755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:20.356615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:20.356615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:20.356615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:20.356615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:20.356615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:20.356615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:20.356615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:20.356615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:20.356615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:20.356615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:20.356615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:20.356615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:20.361693 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f5bdf6-7931-4b9d-bead-1e0d41aec2f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:20.362238 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 174/345] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:47:20 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:20.363263 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd264020-c73c-40de-a775-42f12f61e8d7 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:20.363969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f5bdf6-7931-4b9d-bead-1e0d41aec2f8 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:20.364453 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f5bdf6-7931-4b9d-bead-1e0d41aec2f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:20.364922 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f5bdf6-7931-4b9d-bead-1e0d41aec2f8 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:20.365825 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f5bdf6-7931-4b9d-bead-1e0d41aec2f8 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:20.366349 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f5bdf6-7931-4b9d-bead-1e0d41aec2f8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:20.367939 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f5bdf6-7931-4b9d-bead-1e0d41aec2f8 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:20.379034 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 172/346] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:20 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:20.380627 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41498c5-c130-468a-9207-602cf57432ab None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:20.381136 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41498c5-c130-468a-9207-602cf57432ab None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:20.381607 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41498c5-c130-468a-9207-602cf57432ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:20.386388 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:20.402079 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-b41498c5-c130-468a-9207-602cf57432ab None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:20.403690 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-b41498c5-c130-468a-9207-602cf57432ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:20.490533 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 173/347] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:20.492580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41498c5-c130-468a-9207-602cf57432ab None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:20.504421 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd264020-c73c-40de-a775-42f12f61e8d7 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:20.598446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:20.602868 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e686a2-bcb1-49da-be70-664dfd92c34b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:20.603455 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e686a2-bcb1-49da-be70-664dfd92c34b None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:20.603821 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e686a2-bcb1-49da-be70-664dfd92c34b None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:20.604000 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e686a2-bcb1-49da-be70-664dfd92c34b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:20.615375 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41498c5-c130-468a-9207-602cf57432ab None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:20.663000 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc9bc78-9aae-46f1-8988-5ff823d4893d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:20.663628 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc9bc78-9aae-46f1-8988-5ff823d4893d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:20.663921 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc9bc78-9aae-46f1-8988-5ff823d4893d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:20.664173 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc9bc78-9aae-46f1-8988-5ff823d4893d None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:20.664842 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc9bc78-9aae-46f1-8988-5ff823d4893d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:20.665142 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc9bc78-9aae-46f1-8988-5ff823d4893d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:20.666560 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc9bc78-9aae-46f1-8988-5ff823d4893d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:20.683287 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 174/348] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:20 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:20.684071 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc9bc78-9aae-46f1-8988-5ff823d4893d None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:20.756251 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e686a2-bcb1-49da-be70-664dfd92c34b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:20.757728 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e686a2-bcb1-49da-be70-664dfd92c34b None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:20.787913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:20.876899 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc9bc78-9aae-46f1-8988-5ff823d4893d None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:20.919098 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7540911-8ae5-429b-8301-f73d968ced23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:20.920096 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7540911-8ae5-429b-8301-f73d968ced23 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:20.921659 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7540911-8ae5-429b-8301-f73d968ced23 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:20.922183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7540911-8ae5-429b-8301-f73d968ced23 None admin] PATH_INFO: `/v3/users/ea0f0e12b27343b4808a542e80985a93` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:20.922807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 175/349] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 560 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:20.923664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e686a2-bcb1-49da-be70-664dfd92c34b None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:20.924050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7540911-8ae5-429b-8301-f73d968ced23 None admin] RBAC: Authorizing `identity:get_user(user_id=ea0f0e12b27343b4808a542e80985a93)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:20.925806 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7540911-8ae5-429b-8301-f73d968ced23 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:20.927859 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 175/350] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:47:20 2022] GET /identity/v3/users/ea0f0e12b27343b4808a542e80985a93 => generated 293 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:20.928840 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7540911-8ae5-429b-8301-f73d968ced23 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:20.951644 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:21.058777 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e686a2-bcb1-49da-be70-664dfd92c34b None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:21.061115 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7540911-8ae5-429b-8301-f73d968ced23 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:21.085993 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:21.102787 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5095a43f-4649-4ca0-b608-7d670ff47e0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:21.103447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5095a43f-4649-4ca0-b608-7d670ff47e0f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:21.103725 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5095a43f-4649-4ca0-b608-7d670ff47e0f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:21.104002 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5095a43f-4649-4ca0-b608-7d670ff47e0f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:21.105615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a313b16-e77c-4fc4-aebe-5b60831f86d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:21.106263 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a313b16-e77c-4fc4-aebe-5b60831f86d6 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:21.106543 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a313b16-e77c-4fc4-aebe-5b60831f86d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:21.107099 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a313b16-e77c-4fc4-aebe-5b60831f86d6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:21.116489 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5095a43f-4649-4ca0-b608-7d670ff47e0f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:21.122020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-5095a43f-4649-4ca0-b608-7d670ff47e0f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:21.122020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:21.122020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:21.122020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:21.122020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:21.122020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:21.122020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:21.122020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:21.122020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:21.122020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:21.122020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:21.122020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:21.122020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:21.122020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:21.122020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:21.122020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:21.122954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:21.122954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:21.122954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:21.122954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:21.122954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:21.122954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:21.122954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:21.122954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:21.122954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:21.122954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:21.122954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:21.122954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:21.122954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:21.122954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:21.122954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:21.122954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:21.123723 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:21.123723 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:21.123723 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:21.123723 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:21.123723 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:21.123723 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:21.123723 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:21.123723 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:21.123723 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:21.123723 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:21.123723 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:21.123723 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:21.126481 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 176/351] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:20 2022] GET /identity/v3/roles/member => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:21.127154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5095a43f-4649-4ca0-b608-7d670ff47e0f None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:21.190517 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a313b16-e77c-4fc4-aebe-5b60831f86d6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:21.191990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a313b16-e77c-4fc4-aebe-5b60831f86d6 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:21.229773 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:21.264171 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5095a43f-4649-4ca0-b608-7d670ff47e0f None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:21.311245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5b85a0-eb2e-422b-a56a-d22464d964d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:21.311245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b85a0-eb2e-422b-a56a-d22464d964d4 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:21.311245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b85a0-eb2e-422b-a56a-d22464d964d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:21.312051 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b85a0-eb2e-422b-a56a-d22464d964d4 None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:21.313161 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5b85a0-eb2e-422b-a56a-d22464d964d4 None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:21.314693 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5b85a0-eb2e-422b-a56a-d22464d964d4 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:21.316675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 177/352] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:21 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:21.321260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71432bda-e934-4883-87f0-6fc6272fe385 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:21.321260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71432bda-e934-4883-87f0-6fc6272fe385 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:21.321260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71432bda-e934-4883-87f0-6fc6272fe385 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:21.321260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 178/353] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:21 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:21.321260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71432bda-e934-4883-87f0-6fc6272fe385 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:21.326870 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 176/354] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:21.328329 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a313b16-e77c-4fc4-aebe-5b60831f86d6 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:21.349030 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:21.454564 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71432bda-e934-4883-87f0-6fc6272fe385 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:21.464213 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a313b16-e77c-4fc4-aebe-5b60831f86d6 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:21.503604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac28faa-2cae-443f-a6d8-1408ac6ac5bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:21.504249 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac28faa-2cae-443f-a6d8-1408ac6ac5bb None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:21.504529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac28faa-2cae-443f-a6d8-1408ac6ac5bb None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:21.504802 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac28faa-2cae-443f-a6d8-1408ac6ac5bb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:21.505554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac28faa-2cae-443f-a6d8-1408ac6ac5bb None admin] RBAC: Adding query filter params (role.id=e8be279b370349c8a5b9a06dd763e1b5, user.id=ea0f0e12b27343b4808a542e80985a93, scope.system=all) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:21.505863 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac28faa-2cae-443f-a6d8-1408ac6ac5bb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:21.507388 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac28faa-2cae-443f-a6d8-1408ac6ac5bb None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:21.512684 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ec532a-696a-443b-bcc4-4daf98cb4bb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:21.514296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ec532a-696a-443b-bcc4-4daf98cb4bb7 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:21.514296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ec532a-696a-443b-bcc4-4daf98cb4bb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:21.514296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ec532a-696a-443b-bcc4-4daf98cb4bb7 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:21.515189 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ec532a-696a-443b-bcc4-4daf98cb4bb7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:21.515402 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ec532a-696a-443b-bcc4-4daf98cb4bb7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:21.517064 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ec532a-696a-443b-bcc4-4daf98cb4bb7 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:21.524390 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 179/355] 10.222.0.27 () {56 vars in 1308 bytes} [Wed Sep 7 03:47:21 2022] GET /identity/v3/role_assignments?user.id=ea0f0e12b27343b4808a542e80985a93&scope.system=all&role.id=e8be279b370349c8a5b9a06dd763e1b5 => generated 510 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:21.525420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac28faa-2cae-443f-a6d8-1408ac6ac5bb None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:21.527502 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 177/356] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:21.528775 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cf8d12-0435-4cfa-9727-6d3bd0711cc3 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:21.529058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cf8d12-0435-4cfa-9727-6d3bd0711cc3 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:21.529328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cf8d12-0435-4cfa-9727-6d3bd0711cc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:21.530958 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:21.549961 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-30cf8d12-0435-4cfa-9727-6d3bd0711cc3 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:21.552314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-30cf8d12-0435-4cfa-9727-6d3bd0711cc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:21.638210 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 178/357] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:21.638210 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cf8d12-0435-4cfa-9727-6d3bd0711cc3 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:21.650373 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:21.650922 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac28faa-2cae-443f-a6d8-1408ac6ac5bb None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:21.700105 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a020bb-c4fb-48b2-a798-52a3e9d4d853 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:21.700944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a020bb-c4fb-48b2-a798-52a3e9d4d853 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:21.701287 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a020bb-c4fb-48b2-a798-52a3e9d4d853 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:21.701676 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a020bb-c4fb-48b2-a798-52a3e9d4d853 None admin] PATH_INFO: `/v3/groups/cae75895d7c9487494f7ba61ceeb3d10` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:21.709622 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a020bb-c4fb-48b2-a798-52a3e9d4d853 None admin] RBAC: Authorizing `identity:get_group(group_id=cae75895d7c9487494f7ba61ceeb3d10)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:21.711175 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a020bb-c4fb-48b2-a798-52a3e9d4d853 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:21.713564 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 180/358] 10.222.0.27 () {56 vars in 1180 bytes} [Wed Sep 7 03:47:21 2022] GET /identity/v3/groups/cae75895d7c9487494f7ba61ceeb3d10 => generated 226 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:21.714472 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a020bb-c4fb-48b2-a798-52a3e9d4d853 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:21.764245 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cf8d12-0435-4cfa-9727-6d3bd0711cc3 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:21.797999 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:21.819947 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2ec568-a292-42f0-b74f-5df229f3fd3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:21.820694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2ec568-a292-42f0-b74f-5df229f3fd3c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:21.820979 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2ec568-a292-42f0-b74f-5df229f3fd3c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:21.821275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2ec568-a292-42f0-b74f-5df229f3fd3c None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:21.822539 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2ec568-a292-42f0-b74f-5df229f3fd3c None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:21.823969 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2ec568-a292-42f0-b74f-5df229f3fd3c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:21.825876 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 179/359] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:21 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:21.826914 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35eea4d-955a-4ea2-8183-4679a995b443 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:21.827247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35eea4d-955a-4ea2-8183-4679a995b443 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:21.827614 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35eea4d-955a-4ea2-8183-4679a995b443 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:21.829199 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 180/360] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:21.829912 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35eea4d-955a-4ea2-8183-4679a995b443 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:21.844788 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a020bb-c4fb-48b2-a798-52a3e9d4d853 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:21.889437 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243dabea-dfb8-41f1-bf00-6f3f4e32628f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:21.889437 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243dabea-dfb8-41f1-bf00-6f3f4e32628f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:21.890334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243dabea-dfb8-41f1-bf00-6f3f4e32628f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:21.890621 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243dabea-dfb8-41f1-bf00-6f3f4e32628f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:21.925993 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:21.955304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35eea4d-955a-4ea2-8183-4679a995b443 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:21.971446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243dabea-dfb8-41f1-bf00-6f3f4e32628f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:21.972857 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243dabea-dfb8-41f1-bf00-6f3f4e32628f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:21.998540 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1acfe85-12b0-4738-b2f2-c0b117ec8519 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:21.999233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1acfe85-12b0-4738-b2f2-c0b117ec8519 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:21.999513 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1acfe85-12b0-4738-b2f2-c0b117ec8519 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:21.999836 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1acfe85-12b0-4738-b2f2-c0b117ec8519 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:22.000552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1acfe85-12b0-4738-b2f2-c0b117ec8519 None admin] RBAC: Adding query filter params (group.id=cae75895d7c9487494f7ba61ceeb3d10, role.id=34235948f8f943a99328516c2171cfff, scope.project.id=d6715600d626423b8e8a06d45bb53d08) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:22.000856 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1acfe85-12b0-4738-b2f2-c0b117ec8519 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:22.002309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1acfe85-12b0-4738-b2f2-c0b117ec8519 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:22.009467 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 181/361] 10.222.0.27 () {56 vars in 1376 bytes} [Wed Sep 7 03:47:21 2022] GET /identity/v3/role_assignments?group.id=cae75895d7c9487494f7ba61ceeb3d10&scope.project.id=d6715600d626423b8e8a06d45bb53d08&role.id=34235948f8f943a99328516c2171cfff => generated 260 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:22.009604 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1acfe85-12b0-4738-b2f2-c0b117ec8519 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:22.059253 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:22.121357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 181/362] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:22.123730 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14259a4-13f9-4fb5-b5e8-24e00c9e3e47 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:22.124576 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14259a4-13f9-4fb5-b5e8-24e00c9e3e47 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:22.124851 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14259a4-13f9-4fb5-b5e8-24e00c9e3e47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:22.147329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-d14259a4-13f9-4fb5-b5e8-24e00c9e3e47 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:22.149954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-d14259a4-13f9-4fb5-b5e8-24e00c9e3e47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:22.174186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1acfe85-12b0-4738-b2f2-c0b117ec8519 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:22.220968 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f0c0b8-27d0-46b3-a29f-1c625142a80c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:22.221663 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f0c0b8-27d0-46b3-a29f-1c625142a80c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:22.221888 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f0c0b8-27d0-46b3-a29f-1c625142a80c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:22.222184 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f0c0b8-27d0-46b3-a29f-1c625142a80c None admin] PATH_INFO: `/v3/groups/cae75895d7c9487494f7ba61ceeb3d10` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:22.223309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f0c0b8-27d0-46b3-a29f-1c625142a80c None admin] RBAC: Authorizing `identity:get_group(group_id=cae75895d7c9487494f7ba61ceeb3d10)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:22.224729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f0c0b8-27d0-46b3-a29f-1c625142a80c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:22.231217 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 182/363] 10.222.0.27 () {56 vars in 1180 bytes} [Wed Sep 7 03:47:22 2022] GET /identity/v3/groups/cae75895d7c9487494f7ba61ceeb3d10 => generated 226 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:22.232096 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f0c0b8-27d0-46b3-a29f-1c625142a80c None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:22.249157 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:22.252760 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 182/364] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:22.253824 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14259a4-13f9-4fb5-b5e8-24e00c9e3e47 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:22.351848 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f0c0b8-27d0-46b3-a29f-1c625142a80c None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:22.377792 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:22.382675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14259a4-13f9-4fb5-b5e8-24e00c9e3e47 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:22.391744 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a36420-5d72-4fe1-8e29-4989601073bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:22.392672 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a36420-5d72-4fe1-8e29-4989601073bc None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:22.393149 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a36420-5d72-4fe1-8e29-4989601073bc None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:22.393672 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a36420-5d72-4fe1-8e29-4989601073bc None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:22.406176 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a36420-5d72-4fe1-8e29-4989601073bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:22.412714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-17a36420-5d72-4fe1-8e29-4989601073bc None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:22.412714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:22.412714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:22.412714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:22.412714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:22.412714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:22.412714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:22.412714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:22.412714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:22.412714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:22.412714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:22.412714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:22.412714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:22.412714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:22.412714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:22.412714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:22.413588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:22.413588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:22.413588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:22.413588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:22.413588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:22.413588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:22.413588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:22.413588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:22.413588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:22.413588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:22.413588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:22.413588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:22.413588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:22.413588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:22.413588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:22.413588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:22.414451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:22.414451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:22.414451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:22.414451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:22.414451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:22.414451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:22.414451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:22.414451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:22.414451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:22.414451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:22.414451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:22.414451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:22.416754 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 183/365] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:47:22 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:22.417948 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a36420-5d72-4fe1-8e29-4989601073bc None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:22.424987 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4c74bc-31ea-4e09-b7fd-59178dc21681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:22.425738 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4c74bc-31ea-4e09-b7fd-59178dc21681 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:22.426169 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4c74bc-31ea-4e09-b7fd-59178dc21681 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:22.426497 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4c74bc-31ea-4e09-b7fd-59178dc21681 None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:22.428136 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4c74bc-31ea-4e09-b7fd-59178dc21681 None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:22.429959 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4c74bc-31ea-4e09-b7fd-59178dc21681 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:22.433216 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 183/366] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:22 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:22.433216 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4c74bc-31ea-4e09-b7fd-59178dc21681 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:22.494561 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:22.534067 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a36420-5d72-4fe1-8e29-4989601073bc None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:22.544326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4c74bc-31ea-4e09-b7fd-59178dc21681 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:22.573527 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4887aa9e-c7c9-40fb-8838-13012c3f2d4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:22.574446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4887aa9e-c7c9-40fb-8838-13012c3f2d4d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:22.574900 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4887aa9e-c7c9-40fb-8838-13012c3f2d4d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:22.575317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4887aa9e-c7c9-40fb-8838-13012c3f2d4d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:22.576299 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4887aa9e-c7c9-40fb-8838-13012c3f2d4d None admin] RBAC: Adding query filter params (group.id=cae75895d7c9487494f7ba61ceeb3d10, role.id=285107fb9b2449fe9f1d29f3075a246f, scope.project.id=d6715600d626423b8e8a06d45bb53d08) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:22.576751 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4887aa9e-c7c9-40fb-8838-13012c3f2d4d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:22.579195 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4887aa9e-c7c9-40fb-8838-13012c3f2d4d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:22.584872 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f40a70d-e969-40bd-8588-33abe3c95fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:22.585107 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f40a70d-e969-40bd-8588-33abe3c95fba None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:22.587109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 184/367] 10.222.0.27 () {56 vars in 1376 bytes} [Wed Sep 7 03:47:22 2022] GET /identity/v3/role_assignments?group.id=cae75895d7c9487494f7ba61ceeb3d10&scope.project.id=d6715600d626423b8e8a06d45bb53d08&role.id=285107fb9b2449fe9f1d29f3075a246f => generated 260 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:22.587759 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f40a70d-e969-40bd-8588-33abe3c95fba None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:22.588128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f40a70d-e969-40bd-8588-33abe3c95fba None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:22.588656 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4887aa9e-c7c9-40fb-8838-13012c3f2d4d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:22.588938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f40a70d-e969-40bd-8588-33abe3c95fba None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:22.589256 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f40a70d-e969-40bd-8588-33abe3c95fba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:22.591763 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f40a70d-e969-40bd-8588-33abe3c95fba None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:22.601109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 184/368] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:22 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:22.601983 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f40a70d-e969-40bd-8588-33abe3c95fba None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:22.646538 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:22.739392 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4887aa9e-c7c9-40fb-8838-13012c3f2d4d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:22.740393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f40a70d-e969-40bd-8588-33abe3c95fba None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:22.774903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:22.782793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bf8db2-5270-4e3d-8692-f219779151b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:22.783792 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bf8db2-5270-4e3d-8692-f219779151b8 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:22.784275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bf8db2-5270-4e3d-8692-f219779151b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:22.784552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bf8db2-5270-4e3d-8692-f219779151b8 None admin] PATH_INFO: `/v3/groups/cae75895d7c9487494f7ba61ceeb3d10` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:22.785681 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8bf8db2-5270-4e3d-8692-f219779151b8 None admin] RBAC: Authorizing `identity:get_group(group_id=cae75895d7c9487494f7ba61ceeb3d10)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:22.787164 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8bf8db2-5270-4e3d-8692-f219779151b8 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:22.788979 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 185/369] 10.222.0.27 () {56 vars in 1180 bytes} [Wed Sep 7 03:47:22 2022] GET /identity/v3/groups/cae75895d7c9487494f7ba61ceeb3d10 => generated 226 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:22.789606 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bf8db2-5270-4e3d-8692-f219779151b8 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:22.791700 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4a2824-13f8-4f89-ad89-842f32a5e9e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:22.792559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4a2824-13f8-4f89-ad89-842f32a5e9e1 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:22.792835 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4a2824-13f8-4f89-ad89-842f32a5e9e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:22.793992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4a2824-13f8-4f89-ad89-842f32a5e9e1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:22.810024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4a2824-13f8-4f89-ad89-842f32a5e9e1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:22.817525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-3a4a2824-13f8-4f89-ad89-842f32a5e9e1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:47:22.817525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:22.817525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:22.817525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:22.817525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:22.817525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:22.817525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:22.817525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:22.817525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:22.817525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:22.817525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:22.817525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:22.817525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:22.817525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:22.817525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:22.817525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:22.818562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:22.818562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:22.818562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:22.818562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:22.818562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:22.818562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:22.818562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:22.818562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:22.818562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:22.818562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:22.818562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:22.818562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:22.818562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:22.818562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:22.818562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:22.818562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:22.819365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:22.819365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:22.819365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:22.819365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:22.819365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:22.819365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:22.819365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:22.819365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:22.819365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:22.819365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:22.819365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:47:22.819365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:22.819365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 185/370] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:47:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:22.820011 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4a2824-13f8-4f89-ad89-842f32a5e9e1 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:22.908790 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bf8db2-5270-4e3d-8692-f219779151b8 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:22.926485 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:22.943434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4a2824-13f8-4f89-ad89-842f32a5e9e1 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:22.948815 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8345c653-7351-4af9-b71b-38b4a6ad7e94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:22.949434 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8345c653-7351-4af9-b71b-38b4a6ad7e94 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:22.949712 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8345c653-7351-4af9-b71b-38b4a6ad7e94 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:22.949987 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8345c653-7351-4af9-b71b-38b4a6ad7e94 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:22.981425 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840bc7b8-ee01-498f-9aaa-3054f4fd5d94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:22.982040 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840bc7b8-ee01-498f-9aaa-3054f4fd5d94 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:22.982372 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840bc7b8-ee01-498f-9aaa-3054f4fd5d94 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:22.982648 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840bc7b8-ee01-498f-9aaa-3054f4fd5d94 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:22.983999 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840bc7b8-ee01-498f-9aaa-3054f4fd5d94 None admin] RBAC: Adding query filter params (group.id=cae75895d7c9487494f7ba61ceeb3d10, role.id=34235948f8f943a99328516c2171cfff, scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:22.984532 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840bc7b8-ee01-498f-9aaa-3054f4fd5d94 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:22.986177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840bc7b8-ee01-498f-9aaa-3054f4fd5d94 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:22.992788 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 186/371] 10.222.0.27 () {56 vars in 1376 bytes} [Wed Sep 7 03:47:22 2022] GET /identity/v3/role_assignments?group.id=cae75895d7c9487494f7ba61ceeb3d10&scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54&role.id=34235948f8f943a99328516c2171cfff => generated 260 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:22.994087 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840bc7b8-ee01-498f-9aaa-3054f4fd5d94 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:23.043783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:23.050247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8345c653-7351-4af9-b71b-38b4a6ad7e94 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:23.051928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8345c653-7351-4af9-b71b-38b4a6ad7e94 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:23.132402 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840bc7b8-ee01-498f-9aaa-3054f4fd5d94 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:23.169545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c446342-c3fe-4f5e-bced-ae982758f08a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:23.170200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c446342-c3fe-4f5e-bced-ae982758f08a None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:23.170485 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c446342-c3fe-4f5e-bced-ae982758f08a None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:23.170875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c446342-c3fe-4f5e-bced-ae982758f08a None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:23.171663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c446342-c3fe-4f5e-bced-ae982758f08a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:23.172028 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c446342-c3fe-4f5e-bced-ae982758f08a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:23.173449 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c446342-c3fe-4f5e-bced-ae982758f08a None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:23.175594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 186/372] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:23.181251 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8345c653-7351-4af9-b71b-38b4a6ad7e94 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:23.183557 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 187/373] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:23.193094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c446342-c3fe-4f5e-bced-ae982758f08a None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:23.214188 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:23.322821 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c446342-c3fe-4f5e-bced-ae982758f08a None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:23.330648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:23.334139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8345c653-7351-4af9-b71b-38b4a6ad7e94 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:23.362283 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc89de7-cda7-4065-8d5e-ba5ed36e23be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:23.362905 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc89de7-cda7-4065-8d5e-ba5ed36e23be None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:23.363183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc89de7-cda7-4065-8d5e-ba5ed36e23be None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:23.363454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc89de7-cda7-4065-8d5e-ba5ed36e23be None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:23.378565 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4551f92-34d3-4346-892a-016084e49299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:23.379314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4551f92-34d3-4346-892a-016084e49299 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:23.379600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4551f92-34d3-4346-892a-016084e49299 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:23.379878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4551f92-34d3-4346-892a-016084e49299 None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:23.383225 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4551f92-34d3-4346-892a-016084e49299 None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:23.384782 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4551f92-34d3-4346-892a-016084e49299 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:23.386893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 187/374] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:23 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:23.393097 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4551f92-34d3-4346-892a-016084e49299 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:23.441608 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc89de7-cda7-4065-8d5e-ba5ed36e23be None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:23.443133 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc89de7-cda7-4065-8d5e-ba5ed36e23be None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:23.459285 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:23.555885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 188/375] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 07 03:47:23.562101 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc89de7-cda7-4065-8d5e-ba5ed36e23be None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:23.584177 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4551f92-34d3-4346-892a-016084e49299 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:23.676186 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc89de7-cda7-4065-8d5e-ba5ed36e23be None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:23.704779 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:23.715911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2e27eb-46d8-4c00-84cb-8762d53b6871 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:23.716637 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2e27eb-46d8-4c00-84cb-8762d53b6871 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:23.716923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2e27eb-46d8-4c00-84cb-8762d53b6871 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:23.717200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2e27eb-46d8-4c00-84cb-8762d53b6871 None admin] PATH_INFO: `/v3/projects/388f90b795764a26a3cfac5c43c50a76` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:23.718924 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2e27eb-46d8-4c00-84cb-8762d53b6871 None admin] RBAC: Authorizing `identity:get_project(project_id=388f90b795764a26a3cfac5c43c50a76)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:23.720403 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2e27eb-46d8-4c00-84cb-8762d53b6871 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:23.722053 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 189/376] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:23 2022] GET /identity/v3/projects/388f90b795764a26a3cfac5c43c50a76 => generated 344 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:23.723106 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89b2af8-77c5-454b-8b28-3d8d9c7f9355 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:23.723762 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89b2af8-77c5-454b-8b28-3d8d9c7f9355 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:23.724039 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89b2af8-77c5-454b-8b28-3d8d9c7f9355 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:23.724308 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89b2af8-77c5-454b-8b28-3d8d9c7f9355 None admin] PATH_INFO: `/v3/groups/cae75895d7c9487494f7ba61ceeb3d10` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:23.727450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2e27eb-46d8-4c00-84cb-8762d53b6871 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:23.730829 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89b2af8-77c5-454b-8b28-3d8d9c7f9355 None admin] RBAC: Authorizing `identity:get_group(group_id=cae75895d7c9487494f7ba61ceeb3d10)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:23.732296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89b2af8-77c5-454b-8b28-3d8d9c7f9355 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:23.734788 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 188/377] 10.222.0.27 () {56 vars in 1180 bytes} [Wed Sep 7 03:47:23 2022] GET /identity/v3/groups/cae75895d7c9487494f7ba61ceeb3d10 => generated 226 bytes in 342 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:23.743065 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89b2af8-77c5-454b-8b28-3d8d9c7f9355 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:23.802514 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:23.851821 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2e27eb-46d8-4c00-84cb-8762d53b6871 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:23.892734 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48414734-9eae-42b1-adac-efef4b0aff17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:23.893390 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48414734-9eae-42b1-adac-efef4b0aff17 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:23.893668 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48414734-9eae-42b1-adac-efef4b0aff17 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:23.893942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48414734-9eae-42b1-adac-efef4b0aff17 None admin] PATH_INFO: `/v3/groups/7c25c11e1fdd4d1481b40cb29666e410` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:23.900652 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48414734-9eae-42b1-adac-efef4b0aff17 None admin] RBAC: Authorizing `identity:get_group(group_id=7c25c11e1fdd4d1481b40cb29666e410)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:23.902200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48414734-9eae-42b1-adac-efef4b0aff17 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:23.904545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 190/378] 10.222.0.27 () {56 vars in 1180 bytes} [Wed Sep 7 03:47:23 2022] GET /identity/v3/groups/7c25c11e1fdd4d1481b40cb29666e410 => generated 229 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:23.913376 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48414734-9eae-42b1-adac-efef4b0aff17 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:23.926523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89b2af8-77c5-454b-8b28-3d8d9c7f9355 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:23.971446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:23.987030 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f324c60-0590-4b85-8149-c7848d7e64b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:23.988003 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f324c60-0590-4b85-8149-c7848d7e64b9 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:23.988558 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f324c60-0590-4b85-8149-c7848d7e64b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:23.989146 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f324c60-0590-4b85-8149-c7848d7e64b9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:23.990084 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f324c60-0590-4b85-8149-c7848d7e64b9 None admin] RBAC: Adding query filter params (group.id=cae75895d7c9487494f7ba61ceeb3d10, role.id=285107fb9b2449fe9f1d29f3075a246f, scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:23.990637 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f324c60-0590-4b85-8149-c7848d7e64b9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:23.992356 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f324c60-0590-4b85-8149-c7848d7e64b9 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:24.006182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 189/379] 10.222.0.27 () {56 vars in 1376 bytes} [Wed Sep 7 03:47:23 2022] GET /identity/v3/role_assignments?group.id=cae75895d7c9487494f7ba61ceeb3d10&scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54&role.id=285107fb9b2449fe9f1d29f3075a246f => generated 260 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:24.033398 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48414734-9eae-42b1-adac-efef4b0aff17 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:24.073552 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ffc803-57f8-498b-8f61-8a8211298df3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:24.074204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ffc803-57f8-498b-8f61-8a8211298df3 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:24.074484 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ffc803-57f8-498b-8f61-8a8211298df3 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:24.074753 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ffc803-57f8-498b-8f61-8a8211298df3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:24.075497 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ffc803-57f8-498b-8f61-8a8211298df3 None admin] RBAC: Adding query filter params (group.id=7c25c11e1fdd4d1481b40cb29666e410, role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=388f90b795764a26a3cfac5c43c50a76) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:24.075801 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ffc803-57f8-498b-8f61-8a8211298df3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:24.077576 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ffc803-57f8-498b-8f61-8a8211298df3 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:24.084823 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 191/380] 10.222.0.27 () {56 vars in 1376 bytes} [Wed Sep 7 03:47:23 2022] GET /identity/v3/role_assignments?group.id=7c25c11e1fdd4d1481b40cb29666e410&scope.project.id=388f90b795764a26a3cfac5c43c50a76&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 260 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:24.675100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed851ad-c97d-4c77-9909-d055f47e1a10 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:24.675636 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed851ad-c97d-4c77-9909-d055f47e1a10 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:24.675636 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed851ad-c97d-4c77-9909-d055f47e1a10 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:24.677152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 190/381] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:24.682408 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5114fcf5-ca9a-4129-937d-bf2e9573978b None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:24.682824 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5114fcf5-ca9a-4129-937d-bf2e9573978b None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:24.682990 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5114fcf5-ca9a-4129-937d-bf2e9573978b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:24.704081 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-5114fcf5-ca9a-4129-937d-bf2e9573978b None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:24.705292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-5114fcf5-ca9a-4129-937d-bf2e9573978b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:24.795968 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 192/382] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:25.154463 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed851ad-c97d-4c77-9909-d055f47e1a10 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:25.289395 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:25.291470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed851ad-c97d-4c77-9909-d055f47e1a10 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:25.337125 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5197d16b-125f-423e-b425-b389996a466c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:25.337755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5197d16b-125f-423e-b425-b389996a466c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:25.338029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5197d16b-125f-423e-b425-b389996a466c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:25.338330 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5197d16b-125f-423e-b425-b389996a466c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:25.350968 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5197d16b-125f-423e-b425-b389996a466c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:25.355882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-5197d16b-125f-423e-b425-b389996a466c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:25.355882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:25.355882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:25.355882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:25.355882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:25.355882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:25.355882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:25.355882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:25.355882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:25.355882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:25.355882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:25.355882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:25.355882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:25.355882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:25.355882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:25.355882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:25.356809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:25.356809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:25.356809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:25.356809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:25.356809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:25.356809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:25.356809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:25.356809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:25.356809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:25.356809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:25.356809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:25.356809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:25.356809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:25.356809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:25.356809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:25.356809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:25.357652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:25.357652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:25.357652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:25.357652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:25.357652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:25.357652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:25.357652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:25.357652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:25.357652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:25.357652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:25.357652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:25.357652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:25.357652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 191/383] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:25 2022] GET /identity/v3/roles/member => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:25.364687 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5114fcf5-ca9a-4129-937d-bf2e9573978b None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:25.412081 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c2d2d5-20eb-42bc-ab94-ea4d8262cc60 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:25.412369 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c2d2d5-20eb-42bc-ab94-ea4d8262cc60 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:25.412523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c2d2d5-20eb-42bc-ab94-ea4d8262cc60 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:25.415490 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 192/384] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:47:25.424304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4668c80e-909b-4a31-a055-3a69fc43aadb None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:25.424899 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4668c80e-909b-4a31-a055-3a69fc43aadb None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:25.425184 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4668c80e-909b-4a31-a055-3a69fc43aadb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:25.447226 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:25.462584 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-4668c80e-909b-4a31-a055-3a69fc43aadb None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:25.463822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-4668c80e-909b-4a31-a055-3a69fc43aadb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:25.564074 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5114fcf5-ca9a-4129-937d-bf2e9573978b None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:25.607437 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 193/385] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:25.630831 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f339e7-eec3-4d1e-8226-f11488a19a19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:25.631457 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f339e7-eec3-4d1e-8226-f11488a19a19 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:25.631736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f339e7-eec3-4d1e-8226-f11488a19a19 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:25.632012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f339e7-eec3-4d1e-8226-f11488a19a19 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:25.632691 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f339e7-eec3-4d1e-8226-f11488a19a19 None admin] RBAC: Adding query filter params (name=member) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:25.632995 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f339e7-eec3-4d1e-8226-f11488a19a19 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:25.634405 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f339e7-eec3-4d1e-8226-f11488a19a19 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:25.649598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 193/386] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:25.655813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4668c80e-909b-4a31-a055-3a69fc43aadb None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:25.790624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:25.799649 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4668c80e-909b-4a31-a055-3a69fc43aadb None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:25.838570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b19b3a4-6677-4f2c-88e8-0dc141c71aa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:25.839199 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b19b3a4-6677-4f2c-88e8-0dc141c71aa4 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:25.839474 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b19b3a4-6677-4f2c-88e8-0dc141c71aa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:25.839741 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b19b3a4-6677-4f2c-88e8-0dc141c71aa4 None admin] PATH_INFO: `/v3/groups/cae75895d7c9487494f7ba61ceeb3d10` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:25.840845 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b19b3a4-6677-4f2c-88e8-0dc141c71aa4 None admin] RBAC: Authorizing `identity:get_group(group_id=cae75895d7c9487494f7ba61ceeb3d10)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:25.842298 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b19b3a4-6677-4f2c-88e8-0dc141c71aa4 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:25.844451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 194/387] 10.222.0.27 () {56 vars in 1180 bytes} [Wed Sep 7 03:47:25 2022] GET /identity/v3/groups/cae75895d7c9487494f7ba61ceeb3d10 => generated 226 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:25.849849 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f339e7-eec3-4d1e-8226-f11488a19a19 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:25.951210 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b19b3a4-6677-4f2c-88e8-0dc141c71aa4 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:26.024052 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f339e7-eec3-4d1e-8226-f11488a19a19 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:26.025850 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:26.084399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b19b3a4-6677-4f2c-88e8-0dc141c71aa4 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:26.095160 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27956bf6-5261-4b2e-972e-a0b27aa7f6b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:26.096118 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27956bf6-5261-4b2e-972e-a0b27aa7f6b6 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:26.096632 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27956bf6-5261-4b2e-972e-a0b27aa7f6b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:26.097108 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27956bf6-5261-4b2e-972e-a0b27aa7f6b6 None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:26.098508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27956bf6-5261-4b2e-972e-a0b27aa7f6b6 None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:26.100183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27956bf6-5261-4b2e-972e-a0b27aa7f6b6 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:26.102327 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 194/388] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:25 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:26.103684 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bbf8d4-648d-44db-af16-13c1c319b04b None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:26.104189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bbf8d4-648d-44db-af16-13c1c319b04b None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:26.104673 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bbf8d4-648d-44db-af16-13c1c319b04b None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:26.106344 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 195/389] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:26.107477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bbf8d4-648d-44db-af16-13c1c319b04b None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:26.126807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f15153e-fed3-40f3-9bd6-fa22e3d0b24f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:26.127495 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f15153e-fed3-40f3-9bd6-fa22e3d0b24f None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:26.128108 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f15153e-fed3-40f3-9bd6-fa22e3d0b24f None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:26.128386 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f15153e-fed3-40f3-9bd6-fa22e3d0b24f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:26.142789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f15153e-fed3-40f3-9bd6-fa22e3d0b24f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:26.148928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-0f15153e-fed3-40f3-9bd6-fa22e3d0b24f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:26.148928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:26.148928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:26.148928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:26.148928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:26.148928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:26.148928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:26.148928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:26.148928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:26.148928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:26.148928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:26.148928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:26.148928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:26.148928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:26.148928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:26.148928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:26.149794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:26.149794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:26.149794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:26.149794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:26.149794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:26.149794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:26.149794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:26.149794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:26.149794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:26.149794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:26.149794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:26.149794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:26.149794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:26.149794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:26.149794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:26.149794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:26.150685 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:26.150685 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:26.150685 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:26.150685 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:26.150685 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:26.150685 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:26.150685 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:26.150685 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:26.150685 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:26.150685 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:26.150685 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:26.150685 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:26.150685 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 195/390] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:47:25 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:26.151664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ed7dd1-2aeb-486f-b16e-dbd765c96794 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:26.152103 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ed7dd1-2aeb-486f-b16e-dbd765c96794 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:26.152543 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ed7dd1-2aeb-486f-b16e-dbd765c96794 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:26.177245 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-64ed7dd1-2aeb-486f-b16e-dbd765c96794 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:26.179199 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-64ed7dd1-2aeb-486f-b16e-dbd765c96794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:26.198907 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:26.226710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bbf8d4-648d-44db-af16-13c1c319b04b None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:26.272133 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b0e459-de42-4e1c-a536-011f75056edb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:26.273018 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b0e459-de42-4e1c-a536-011f75056edb None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:26.273444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b0e459-de42-4e1c-a536-011f75056edb None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:26.273697 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 196/391] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:26.274102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b0e459-de42-4e1c-a536-011f75056edb None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08/groups/cae75895d7c9487494f7ba61ceeb3d10/roles/34235948f8f943a99328516c2171cfff` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:26.274764 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ed7dd1-2aeb-486f-b16e-dbd765c96794 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:26.276505 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b0e459-de42-4e1c-a536-011f75056edb None admin] RBAC: Authorizing `identity:create_grant(project_id=d6715600d626423b8e8a06d45bb53d08, group_id=cae75895d7c9487494f7ba61ceeb3d10, role_id=34235948f8f943a99328516c2171cfff)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:26.279365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b0e459-de42-4e1c-a536-011f75056edb None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:26.296335 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 196/392] 10.222.0.27 () {58 vars in 1442 bytes} [Wed Sep 7 03:47:26 2022] PUT /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08/groups/cae75895d7c9487494f7ba61ceeb3d10/roles/34235948f8f943a99328516c2171cfff => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:47:26.395899 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ed7dd1-2aeb-486f-b16e-dbd765c96794 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:26.434418 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a92a58-b954-45df-abcd-5583f166d06d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:26.435088 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a92a58-b954-45df-abcd-5583f166d06d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:26.435370 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a92a58-b954-45df-abcd-5583f166d06d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:26.435696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a92a58-b954-45df-abcd-5583f166d06d None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:26.436905 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a92a58-b954-45df-abcd-5583f166d06d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:26.437209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a92a58-b954-45df-abcd-5583f166d06d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:26.438646 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a92a58-b954-45df-abcd-5583f166d06d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:26.447837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 197/393] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:26 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:26.458810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b0e459-de42-4e1c-a536-011f75056edb None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:26.513363 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:26.573981 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a92a58-b954-45df-abcd-5583f166d06d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:26.604350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b0e459-de42-4e1c-a536-011f75056edb None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:26.609543 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:26.643279 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff2fa16-b378-4028-a6e6-40fd4592ecb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:26.644204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff2fa16-b378-4028-a6e6-40fd4592ecb1 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:26.644665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff2fa16-b378-4028-a6e6-40fd4592ecb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:26.645091 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff2fa16-b378-4028-a6e6-40fd4592ecb1 None admin] PATH_INFO: `/v3/groups/cae75895d7c9487494f7ba61ceeb3d10` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:26.646620 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff2fa16-b378-4028-a6e6-40fd4592ecb1 None admin] RBAC: Authorizing `identity:get_group(group_id=cae75895d7c9487494f7ba61ceeb3d10)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:26.648360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff2fa16-b378-4028-a6e6-40fd4592ecb1 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:26.650748 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 197/394] 10.222.0.27 () {56 vars in 1180 bytes} [Wed Sep 7 03:47:26 2022] GET /identity/v3/groups/cae75895d7c9487494f7ba61ceeb3d10 => generated 226 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:26.663220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff2fa16-b378-4028-a6e6-40fd4592ecb1 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:26.715475 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a92a58-b954-45df-abcd-5583f166d06d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:26.753114 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c87551a-de22-42f3-a89a-0263b9af87e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:26.754201 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c87551a-de22-42f3-a89a-0263b9af87e9 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:26.754667 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c87551a-de22-42f3-a89a-0263b9af87e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:26.755104 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c87551a-de22-42f3-a89a-0263b9af87e9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:26.770305 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c87551a-de22-42f3-a89a-0263b9af87e9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:26.776401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-6c87551a-de22-42f3-a89a-0263b9af87e9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:26.776401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:26.776401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:26.776401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:26.776401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:26.776401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:26.776401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:26.776401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:26.776401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:26.776401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:26.776401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:26.776401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:26.776401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:26.776401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:26.776401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:26.776401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:26.777324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:26.777324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:26.777324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:26.777324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:26.777324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:26.777324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:26.777324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:26.777324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:26.777324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:26.777324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:26.777324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:26.777324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:26.777324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:26.777324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:26.777324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:26.777324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:26.778337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:26.778337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:26.778337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:26.778337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:26.778337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:26.778337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:26.778337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:26.778337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:26.778337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:26.778337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:26.778337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:26.778337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:26.778337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 198/395] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:26.779251 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f556c4c-d088-466d-b141-9c6c5d522f9b None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:26.779710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f556c4c-d088-466d-b141-9c6c5d522f9b None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:26.780144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f556c4c-d088-466d-b141-9c6c5d522f9b None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:26.785484 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 199/396] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:26.785484 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f556c4c-d088-466d-b141-9c6c5d522f9b None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:26.801838 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:26.808397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff2fa16-b378-4028-a6e6-40fd4592ecb1 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:26.852955 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651e0e68-140c-48d8-9cc2-b57330c4cb72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:26.853808 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651e0e68-140c-48d8-9cc2-b57330c4cb72 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:26.854090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651e0e68-140c-48d8-9cc2-b57330c4cb72 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:26.855676 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651e0e68-140c-48d8-9cc2-b57330c4cb72 None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:26.857632 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651e0e68-140c-48d8-9cc2-b57330c4cb72 None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:26.859244 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651e0e68-140c-48d8-9cc2-b57330c4cb72 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:26.861567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 198/397] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:26 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:26.862626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37227eea-f207-4684-a604-26d0988f874f None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:26.862908 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37227eea-f207-4684-a604-26d0988f874f None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:26.863188 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37227eea-f207-4684-a604-26d0988f874f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:26.882235 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:26.885118 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-37227eea-f207-4684-a604-26d0988f874f None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:26.886430 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-37227eea-f207-4684-a604-26d0988f874f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:26.928622 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f556c4c-d088-466d-b141-9c6c5d522f9b None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:26.970141 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 199/398] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:26.970141 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37227eea-f207-4684-a604-26d0988f874f None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:26.987908 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d78bfa-4668-4569-a84e-579a620cdb79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:26.988629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d78bfa-4668-4569-a84e-579a620cdb79 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:26.988968 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d78bfa-4668-4569-a84e-579a620cdb79 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:26.989246 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d78bfa-4668-4569-a84e-579a620cdb79 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:26.989923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d78bfa-4668-4569-a84e-579a620cdb79 None admin] RBAC: Adding query filter params (name=member) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:26.990307 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d78bfa-4668-4569-a84e-579a620cdb79 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:26.992788 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d78bfa-4668-4569-a84e-579a620cdb79 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:27.010166 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 200/399] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:27.015904 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d78bfa-4668-4569-a84e-579a620cdb79 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:27.143031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37227eea-f207-4684-a604-26d0988f874f None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:27.154495 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:27.184664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d78bfa-4668-4569-a84e-579a620cdb79 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:27.199609 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5376aa90-8e5e-438c-9b33-fc0aa90d342c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:27.200600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5376aa90-8e5e-438c-9b33-fc0aa90d342c None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:27.201262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5376aa90-8e5e-438c-9b33-fc0aa90d342c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:27.202357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5376aa90-8e5e-438c-9b33-fc0aa90d342c None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08/groups/cae75895d7c9487494f7ba61ceeb3d10/roles/285107fb9b2449fe9f1d29f3075a246f` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:27.205525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5376aa90-8e5e-438c-9b33-fc0aa90d342c None admin] RBAC: Authorizing `identity:create_grant(project_id=d6715600d626423b8e8a06d45bb53d08, group_id=cae75895d7c9487494f7ba61ceeb3d10, role_id=285107fb9b2449fe9f1d29f3075a246f)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:27.207563 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5376aa90-8e5e-438c-9b33-fc0aa90d342c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:27.223807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:27.226681 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 200/400] 10.222.0.27 () {58 vars in 1442 bytes} [Wed Sep 7 03:47:26 2022] PUT /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08/groups/cae75895d7c9487494f7ba61ceeb3d10/roles/285107fb9b2449fe9f1d29f3075a246f => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:47:27.227982 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fa37a0-a9f7-4fee-b3c6-cdba798a27d8 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:27.228285 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fa37a0-a9f7-4fee-b3c6-cdba798a27d8 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:27.228855 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fa37a0-a9f7-4fee-b3c6-cdba798a27d8 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:27.235045 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 201/401] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:27.235263 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b547f119-75c4-4cc2-914f-1ac8f625d7dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:27.235263 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b547f119-75c4-4cc2-914f-1ac8f625d7dc None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:27.235439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b547f119-75c4-4cc2-914f-1ac8f625d7dc None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:27.235757 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b547f119-75c4-4cc2-914f-1ac8f625d7dc None admin] PATH_INFO: `/v3/groups/cae75895d7c9487494f7ba61ceeb3d10` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:27.237903 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96134b9c-dc48-492b-b14a-253474c52754 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:27.238495 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96134b9c-dc48-492b-b14a-253474c52754 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:27.238975 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96134b9c-dc48-492b-b14a-253474c52754 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:27.243281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b547f119-75c4-4cc2-914f-1ac8f625d7dc None admin] RBAC: Authorizing `identity:get_group(group_id=cae75895d7c9487494f7ba61ceeb3d10)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:27.244905 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b547f119-75c4-4cc2-914f-1ac8f625d7dc None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:27.247068 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 201/402] 10.222.0.27 () {56 vars in 1180 bytes} [Wed Sep 7 03:47:27 2022] GET /identity/v3/groups/cae75895d7c9487494f7ba61ceeb3d10 => generated 226 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:27.257349 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b547f119-75c4-4cc2-914f-1ac8f625d7dc None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:27.265439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-96134b9c-dc48-492b-b14a-253474c52754 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:27.266803 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-96134b9c-dc48-492b-b14a-253474c52754 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:27.356128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 202/403] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:47:27.371485 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96134b9c-dc48-492b-b14a-253474c52754 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:27.469537 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b547f119-75c4-4cc2-914f-1ac8f625d7dc None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:27.492222 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:27.512904 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96134b9c-dc48-492b-b14a-253474c52754 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:27.513614 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d276fb1e-980b-4892-9aa2-8241ed8f932c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:27.515300 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d276fb1e-980b-4892-9aa2-8241ed8f932c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:27.516045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d276fb1e-980b-4892-9aa2-8241ed8f932c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:27.516513 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d276fb1e-980b-4892-9aa2-8241ed8f932c None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:27.517881 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d276fb1e-980b-4892-9aa2-8241ed8f932c None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:27.519874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d276fb1e-980b-4892-9aa2-8241ed8f932c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:27.522199 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 202/404] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:27 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:27.528789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d276fb1e-980b-4892-9aa2-8241ed8f932c None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:27.567922 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfb2099-de60-4166-9256-14558bddadeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:27.568598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfb2099-de60-4166-9256-14558bddadeb None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:27.568879 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfb2099-de60-4166-9256-14558bddadeb None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:27.569153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfb2099-de60-4166-9256-14558bddadeb None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:27.581629 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edfb2099-de60-4166-9256-14558bddadeb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:27.586960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-edfb2099-de60-4166-9256-14558bddadeb None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:27.586960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:27.586960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:27.586960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:27.586960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:27.586960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:27.586960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:27.586960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:27.586960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:27.586960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:27.586960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:27.586960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:27.586960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:27.586960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:27.586960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:27.586960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:27.587815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:27.587815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:27.587815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:27.587815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:27.587815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:27.587815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:27.587815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:27.587815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:27.587815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:27.587815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:27.587815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:27.587815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:27.587815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:27.587815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:27.587815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:27.587815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:27.588617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:27.588617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:27.588617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:27.588617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:27.588617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:27.588617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:27.588617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:27.588617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:27.588617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:27.588617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:27.588617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:27.588617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:27.594667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 203/405] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:47:27 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:27.599851 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfb2099-de60-4166-9256-14558bddadeb None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:27.616358 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:27.653834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d276fb1e-980b-4892-9aa2-8241ed8f932c None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:27.693156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72c0c32-523e-429f-8363-2b67cdcd82d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:27.694298 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72c0c32-523e-429f-8363-2b67cdcd82d7 None admin] REQUEST_METHOD: `PUT` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:27.694799 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72c0c32-523e-429f-8363-2b67cdcd82d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:27.695276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72c0c32-523e-429f-8363-2b67cdcd82d7 None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54/groups/cae75895d7c9487494f7ba61ceeb3d10/roles/34235948f8f943a99328516c2171cfff` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:27.697577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72c0c32-523e-429f-8363-2b67cdcd82d7 None admin] RBAC: Authorizing `identity:create_grant(project_id=72fb26d349cf4298b0ad385cd5e2ee54, group_id=cae75895d7c9487494f7ba61ceeb3d10, role_id=34235948f8f943a99328516c2171cfff)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:27.699288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72c0c32-523e-429f-8363-2b67cdcd82d7 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:27.715524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 203/406] 10.222.0.27 () {58 vars in 1442 bytes} [Wed Sep 7 03:47:27 2022] PUT /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54/groups/cae75895d7c9487494f7ba61ceeb3d10/roles/34235948f8f943a99328516c2171cfff => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:47:27.722747 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72c0c32-523e-429f-8363-2b67cdcd82d7 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:27.737787 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfb2099-de60-4166-9256-14558bddadeb None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:27.783668 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a8a9eb-e9e5-4784-b59d-3a33aa4524a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:27.784316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a8a9eb-e9e5-4784-b59d-3a33aa4524a6 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:27.784663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a8a9eb-e9e5-4784-b59d-3a33aa4524a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:27.784943 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a8a9eb-e9e5-4784-b59d-3a33aa4524a6 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:27.785623 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a8a9eb-e9e5-4784-b59d-3a33aa4524a6 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:27.785928 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a8a9eb-e9e5-4784-b59d-3a33aa4524a6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:27.787474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a8a9eb-e9e5-4784-b59d-3a33aa4524a6 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:27.803747 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 204/407] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:27 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:27.805335 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a8a9eb-e9e5-4784-b59d-3a33aa4524a6 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:27.818502 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:27.921672 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a8a9eb-e9e5-4784-b59d-3a33aa4524a6 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:27.933038 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72c0c32-523e-429f-8363-2b67cdcd82d7 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:27.961086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99e1c61-be72-45d8-a560-181e2cf86f33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:27.961755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99e1c61-be72-45d8-a560-181e2cf86f33 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:27.962039 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99e1c61-be72-45d8-a560-181e2cf86f33 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:27.962361 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99e1c61-be72-45d8-a560-181e2cf86f33 None admin] PATH_INFO: `/v3/groups/cae75895d7c9487494f7ba61ceeb3d10` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:27.963629 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99e1c61-be72-45d8-a560-181e2cf86f33 None admin] RBAC: Authorizing `identity:get_group(group_id=cae75895d7c9487494f7ba61ceeb3d10)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:27.965126 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99e1c61-be72-45d8-a560-181e2cf86f33 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:27.970342 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 205/408] 10.222.0.27 () {56 vars in 1180 bytes} [Wed Sep 7 03:47:27 2022] GET /identity/v3/groups/cae75895d7c9487494f7ba61ceeb3d10 => generated 226 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:27.973575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99e1c61-be72-45d8-a560-181e2cf86f33 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:27.995694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:28.010150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa054811-b97c-4bb9-9f8d-7783e61e182d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:28.011190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa054811-b97c-4bb9-9f8d-7783e61e182d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:28.011662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa054811-b97c-4bb9-9f8d-7783e61e182d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:28.012237 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa054811-b97c-4bb9-9f8d-7783e61e182d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:28.025447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa054811-b97c-4bb9-9f8d-7783e61e182d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:28.031265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-fa054811-b97c-4bb9-9f8d-7783e61e182d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:47:28.031265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:28.031265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:28.031265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:28.031265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:28.031265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:28.031265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:28.031265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:28.031265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:28.031265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:28.031265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:28.031265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:28.031265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:28.031265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:28.031265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:28.031265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:28.032147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:28.032147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:28.032147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:28.032147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:28.032147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:28.032147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:28.032147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:28.032147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:28.032147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:28.032147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:28.032147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:28.032147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:28.032147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:28.032147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:28.032147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:28.032147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:28.032952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:28.032952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:28.032952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:28.032952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:28.032952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:28.032952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:28.032952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:28.032952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:28.032952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:28.032952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:28.032952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:47:28.032952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:28.035206 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 204/409] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:47:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:28.041065 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa054811-b97c-4bb9-9f8d-7783e61e182d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:28.065877 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:28.100360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99e1c61-be72-45d8-a560-181e2cf86f33 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:28.142760 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a978a9-a1cf-472d-9c7a-6c6bd3ad1e16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:28.143669 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a978a9-a1cf-472d-9c7a-6c6bd3ad1e16 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:28.144185 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a978a9-a1cf-472d-9c7a-6c6bd3ad1e16 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:28.144665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a978a9-a1cf-472d-9c7a-6c6bd3ad1e16 None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:28.146015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a978a9-a1cf-472d-9c7a-6c6bd3ad1e16 None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:28.147702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a978a9-a1cf-472d-9c7a-6c6bd3ad1e16 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:28.154349 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 206/410] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:27 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:28.155866 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a978a9-a1cf-472d-9c7a-6c6bd3ad1e16 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:28.175267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa054811-b97c-4bb9-9f8d-7783e61e182d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:28.229812 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c2b986-2ee7-4300-857b-5d2257b5d7c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:28.230373 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c2b986-2ee7-4300-857b-5d2257b5d7c3 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:28.230649 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c2b986-2ee7-4300-857b-5d2257b5d7c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:28.230923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c2b986-2ee7-4300-857b-5d2257b5d7c3 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:28.231588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c2b986-2ee7-4300-857b-5d2257b5d7c3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:28.232014 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c2b986-2ee7-4300-857b-5d2257b5d7c3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:28.234703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c2b986-2ee7-4300-857b-5d2257b5d7c3 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:28.246505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 205/411] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:28.251952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c2b986-2ee7-4300-857b-5d2257b5d7c3 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:28.271125 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a978a9-a1cf-472d-9c7a-6c6bd3ad1e16 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:28.317835 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:28.375017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc2d827-ed4d-47ec-9e31-13705189739c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:28.375716 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc2d827-ed4d-47ec-9e31-13705189739c None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:28.375947 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c2b986-2ee7-4300-857b-5d2257b5d7c3 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:28.376055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc2d827-ed4d-47ec-9e31-13705189739c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:28.376268 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc2d827-ed4d-47ec-9e31-13705189739c None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54/groups/cae75895d7c9487494f7ba61ceeb3d10/roles/285107fb9b2449fe9f1d29f3075a246f` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:28.386911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc2d827-ed4d-47ec-9e31-13705189739c None admin] RBAC: Authorizing `identity:create_grant(project_id=72fb26d349cf4298b0ad385cd5e2ee54, group_id=cae75895d7c9487494f7ba61ceeb3d10, role_id=285107fb9b2449fe9f1d29f3075a246f)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:28.388389 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc2d827-ed4d-47ec-9e31-13705189739c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:28.414956 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5431a0-3676-4468-850b-4f4b49c05c28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:28.415581 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5431a0-3676-4468-850b-4f4b49c05c28 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:28.416114 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5431a0-3676-4468-850b-4f4b49c05c28 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:28.416385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5431a0-3676-4468-850b-4f4b49c05c28 None admin] PATH_INFO: `/v3/groups/7c25c11e1fdd4d1481b40cb29666e410` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:28.417534 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5431a0-3676-4468-850b-4f4b49c05c28 None admin] RBAC: Authorizing `identity:get_group(group_id=7c25c11e1fdd4d1481b40cb29666e410)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:28.419405 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5431a0-3676-4468-850b-4f4b49c05c28 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:28.421573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 206/412] 10.222.0.27 () {56 vars in 1180 bytes} [Wed Sep 7 03:47:28 2022] GET /identity/v3/groups/7c25c11e1fdd4d1481b40cb29666e410 => generated 229 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:28.427173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5431a0-3676-4468-850b-4f4b49c05c28 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:28.429236 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 207/413] 10.222.0.27 () {58 vars in 1442 bytes} [Wed Sep 7 03:47:28 2022] PUT /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54/groups/cae75895d7c9487494f7ba61ceeb3d10/roles/285107fb9b2449fe9f1d29f3075a246f => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:47:28.553258 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5431a0-3676-4468-850b-4f4b49c05c28 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:28.573629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:28.605486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5197a0e3-924c-48cd-bd37-3497758391c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:28.606155 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5197a0e3-924c-48cd-bd37-3497758391c6 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:28.606437 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5197a0e3-924c-48cd-bd37-3497758391c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:28.606708 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5197a0e3-924c-48cd-bd37-3497758391c6 None admin] PATH_INFO: `/v3/projects/388f90b795764a26a3cfac5c43c50a76` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:28.610913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5197a0e3-924c-48cd-bd37-3497758391c6 None admin] RBAC: Authorizing `identity:get_project(project_id=388f90b795764a26a3cfac5c43c50a76)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:28.612379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5197a0e3-924c-48cd-bd37-3497758391c6 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:28.614510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 207/414] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:28 2022] GET /identity/v3/projects/388f90b795764a26a3cfac5c43c50a76 => generated 344 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:28.623281 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc2d827-ed4d-47ec-9e31-13705189739c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:28.634675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:28.757559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc2d827-ed4d-47ec-9e31-13705189739c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:28.798315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a04fc5-d44c-4279-a2df-fe7af6c85d7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:28.799347 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a04fc5-d44c-4279-a2df-fe7af6c85d7d None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:28.799832 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a04fc5-d44c-4279-a2df-fe7af6c85d7d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:28.800345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a04fc5-d44c-4279-a2df-fe7af6c85d7d None admin] PATH_INFO: `/v3/projects/388f90b795764a26a3cfac5c43c50a76/groups/7c25c11e1fdd4d1481b40cb29666e410/roles/9d5b19a61c46475d8048b1bd66ca25a1` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:28.802559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a04fc5-d44c-4279-a2df-fe7af6c85d7d None admin] RBAC: Authorizing `identity:create_grant(project_id=388f90b795764a26a3cfac5c43c50a76, group_id=7c25c11e1fdd4d1481b40cb29666e410, role_id=9d5b19a61c46475d8048b1bd66ca25a1)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:28.804310 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a04fc5-d44c-4279-a2df-fe7af6c85d7d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:28.820295 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 208/415] 10.222.0.27 () {58 vars in 1442 bytes} [Wed Sep 7 03:47:28 2022] PUT /identity/v3/projects/388f90b795764a26a3cfac5c43c50a76/groups/7c25c11e1fdd4d1481b40cb29666e410/roles/9d5b19a61c46475d8048b1bd66ca25a1 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:47:29.002030 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80e1f23-673f-4dd9-9048-853a6b408857 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:29.002606 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80e1f23-673f-4dd9-9048-853a6b408857 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:29.002606 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80e1f23-673f-4dd9-9048-853a6b408857 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:29.004257 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 208/416] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:29.010305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55796a1-f0b0-4655-88bb-cfb2e1740ec4 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:29.011237 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55796a1-f0b0-4655-88bb-cfb2e1740ec4 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:29.011737 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55796a1-f0b0-4655-88bb-cfb2e1740ec4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:29.036522 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-c55796a1-f0b0-4655-88bb-cfb2e1740ec4 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:29.037738 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-c55796a1-f0b0-4655-88bb-cfb2e1740ec4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:29.120474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:29.131504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 209/417] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:29.491675 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80e1f23-673f-4dd9-9048-853a6b408857 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:29.620846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80e1f23-673f-4dd9-9048-853a6b408857 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:29.672660 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ff082c-6426-4e93-b71d-2be6002b3c8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:29.673342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ff082c-6426-4e93-b71d-2be6002b3c8a None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:29.673624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ff082c-6426-4e93-b71d-2be6002b3c8a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:29.673898 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ff082c-6426-4e93-b71d-2be6002b3c8a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:29.687897 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ff082c-6426-4e93-b71d-2be6002b3c8a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:29.693204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-b5ff082c-6426-4e93-b71d-2be6002b3c8a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:29.693204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:29.693204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:29.693204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:29.693204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:29.693204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:29.693204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:29.693204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:29.693204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:29.693204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:29.693204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:29.693204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:29.693204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:29.693204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:29.693204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:29.693204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:29.694071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:29.694071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:29.694071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:29.694071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:29.694071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:29.694071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:29.694071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:29.694071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:29.694071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:29.694071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:29.694071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:29.694071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:29.694071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:29.694071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:29.694071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:29.694071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:29.695283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:29.695283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:29.695283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:29.695283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:29.695283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:29.695283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:29.695283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:29.695283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:29.695283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:29.695283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:29.695283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:29.695283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:29.695283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 209/418] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:29.702596 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55796a1-f0b0-4655-88bb-cfb2e1740ec4 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:29.827807 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55796a1-f0b0-4655-88bb-cfb2e1740ec4 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:29.863844 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9240ecb6-5ef9-4069-9795-235f78c120b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:29.864869 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9240ecb6-5ef9-4069-9795-235f78c120b6 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:29.865355 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9240ecb6-5ef9-4069-9795-235f78c120b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:29.865851 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9240ecb6-5ef9-4069-9795-235f78c120b6 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:29.866752 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9240ecb6-5ef9-4069-9795-235f78c120b6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:29.867257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9240ecb6-5ef9-4069-9795-235f78c120b6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:29.868930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9240ecb6-5ef9-4069-9795-235f78c120b6 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:29.882487 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 210/419] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:29.887200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ff082c-6426-4e93-b71d-2be6002b3c8a None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:29.959372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:30.025355 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ff082c-6426-4e93-b71d-2be6002b3c8a None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:30.066196 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7892af-4824-46fe-9abf-22e20df84207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:30.066806 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7892af-4824-46fe-9abf-22e20df84207 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:30.067083 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7892af-4824-46fe-9abf-22e20df84207 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:30.067354 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7892af-4824-46fe-9abf-22e20df84207 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:30.145502 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7892af-4824-46fe-9abf-22e20df84207 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:30.146882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7892af-4824-46fe-9abf-22e20df84207 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:30.158499 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f437999-3044-48a5-8c0e-18934076b764 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:30.159430 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f437999-3044-48a5-8c0e-18934076b764 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:30.159912 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f437999-3044-48a5-8c0e-18934076b764 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:30.163109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 211/420] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:47:30.167360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c94047-2673-4487-a3ae-8f9566e63926 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:30.167642 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c94047-2673-4487-a3ae-8f9566e63926 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:30.168202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c94047-2673-4487-a3ae-8f9566e63926 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:30.203150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-a8c94047-2673-4487-a3ae-8f9566e63926 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:30.204629 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-a8c94047-2673-4487-a3ae-8f9566e63926 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:30.229212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:30.283148 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 210/421] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:30.288221 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7892af-4824-46fe-9abf-22e20df84207 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:30.358775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:30.395856 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 212/422] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:47:30.457809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7892af-4824-46fe-9abf-22e20df84207 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:30.506359 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32dcd62-186b-42d7-bb16-f7a1a458bec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:30.507225 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32dcd62-186b-42d7-bb16-f7a1a458bec8 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:30.507675 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32dcd62-186b-42d7-bb16-f7a1a458bec8 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:30.508087 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32dcd62-186b-42d7-bb16-f7a1a458bec8 None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:30.516642 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32dcd62-186b-42d7-bb16-f7a1a458bec8 None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:30.518272 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32dcd62-186b-42d7-bb16-f7a1a458bec8 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:30.520646 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 211/423] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:30 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:30.525770 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c94047-2673-4487-a3ae-8f9566e63926 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:30.536327 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e9235d-708d-47b2-af49-3350dcea8ff8 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:30.538464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e9235d-708d-47b2-af49-3350dcea8ff8 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:30.540017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e9235d-708d-47b2-af49-3350dcea8ff8 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:30.542597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 212/424] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:30 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:30.547420 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33081aa-7829-4d85-94b1-0fbc5b64cea7 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:30.547694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33081aa-7829-4d85-94b1-0fbc5b64cea7 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:30.547967 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33081aa-7829-4d85-94b1-0fbc5b64cea7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:30.574909 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-d33081aa-7829-4d85-94b1-0fbc5b64cea7 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:30.576120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-d33081aa-7829-4d85-94b1-0fbc5b64cea7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:30.589485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:30.688907 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 213/425] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:30.717208 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c94047-2673-4487-a3ae-8f9566e63926 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:30.766476 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423a88fe-12f8-4017-9d26-b0a31b5579b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:30.767125 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423a88fe-12f8-4017-9d26-b0a31b5579b9 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:30.767407 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423a88fe-12f8-4017-9d26-b0a31b5579b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:30.767678 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423a88fe-12f8-4017-9d26-b0a31b5579b9 None admin] PATH_INFO: `/v3/groups/cae75895d7c9487494f7ba61ceeb3d10` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:30.768815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423a88fe-12f8-4017-9d26-b0a31b5579b9 None admin] RBAC: Authorizing `identity:get_group(group_id=cae75895d7c9487494f7ba61ceeb3d10)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:30.770468 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423a88fe-12f8-4017-9d26-b0a31b5579b9 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:30.772332 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 213/426] 10.222.0.27 () {56 vars in 1180 bytes} [Wed Sep 7 03:47:30 2022] GET /identity/v3/groups/cae75895d7c9487494f7ba61ceeb3d10 => generated 226 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:30.778235 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33081aa-7829-4d85-94b1-0fbc5b64cea7 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:30.812740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423a88fe-12f8-4017-9d26-b0a31b5579b9 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:30.917872 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:30.956548 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33081aa-7829-4d85-94b1-0fbc5b64cea7 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:31.000197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423a88fe-12f8-4017-9d26-b0a31b5579b9 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:31.025877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec154cfe-9786-4518-aabd-9742de05e444 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:31.028356 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec154cfe-9786-4518-aabd-9742de05e444 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:31.028634 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec154cfe-9786-4518-aabd-9742de05e444 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:31.034904 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec154cfe-9786-4518-aabd-9742de05e444 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:31.035716 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec154cfe-9786-4518-aabd-9742de05e444 None admin] RBAC: Adding query filter params (group.id=cae75895d7c9487494f7ba61ceeb3d10, role.id=34235948f8f943a99328516c2171cfff, scope.project.id=d6715600d626423b8e8a06d45bb53d08) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:31.036033 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec154cfe-9786-4518-aabd-9742de05e444 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:31.037495 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec154cfe-9786-4518-aabd-9742de05e444 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:31.047606 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 214/427] 10.222.0.27 () {56 vars in 1376 bytes} [Wed Sep 7 03:47:30 2022] GET /identity/v3/role_assignments?group.id=cae75895d7c9487494f7ba61ceeb3d10&scope.project.id=d6715600d626423b8e8a06d45bb53d08&role.id=34235948f8f943a99328516c2171cfff => generated 611 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:31.048735 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec154cfe-9786-4518-aabd-9742de05e444 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:31.069907 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:31.103012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f05f914-c7d6-4db6-bed7-2abeb0eea435 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:31.103741 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f05f914-c7d6-4db6-bed7-2abeb0eea435 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:31.104025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f05f914-c7d6-4db6-bed7-2abeb0eea435 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:31.104303 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f05f914-c7d6-4db6-bed7-2abeb0eea435 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:31.132260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f05f914-c7d6-4db6-bed7-2abeb0eea435 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:31.150038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-6f05f914-c7d6-4db6-bed7-2abeb0eea435 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:31.150038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:31.150038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:31.150038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:31.150038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:31.150038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:31.150038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:31.150038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:31.150038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:31.150038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:31.150038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:31.150038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:31.150038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:31.150038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:31.150038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:31.150038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:31.151022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:31.151022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:31.151022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:31.151022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:31.151022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:31.151022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:31.151022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:31.151022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:31.151022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:31.151022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:31.151022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:31.151022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:31.151022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:31.151022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:31.151022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:31.151022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:31.151893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:31.151893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:31.151893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:31.151893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:31.151893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:31.151893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:31.151893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:31.151893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:31.151893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:31.151893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:31.151893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:31.151893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:31.151893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 214/428] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:47:30 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 339 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:31.160990 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f05f914-c7d6-4db6-bed7-2abeb0eea435 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:31.183706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec154cfe-9786-4518-aabd-9742de05e444 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:31.278227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b913c71c-a46e-4ad7-8a16-9acca7e73a0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:31.283037 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b913c71c-a46e-4ad7-8a16-9acca7e73a0c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:31.283735 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b913c71c-a46e-4ad7-8a16-9acca7e73a0c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:31.284204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b913c71c-a46e-4ad7-8a16-9acca7e73a0c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:31.309462 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b913c71c-a46e-4ad7-8a16-9acca7e73a0c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:31.315194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-b913c71c-a46e-4ad7-8a16-9acca7e73a0c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:31.315194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:31.315194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:31.315194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:31.315194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:31.315194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:31.315194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:31.315194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:31.315194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:31.315194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:31.315194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:31.315194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:31.315194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:31.315194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:31.315194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:31.315194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:31.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:31.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:31.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:31.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:31.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:31.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:31.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:31.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:31.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:31.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:31.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:31.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:31.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:31.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:31.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:31.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:31.316885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:31.316885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:31.316885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:31.316885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:31.316885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:31.316885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:31.316885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:31.316885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:31.316885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:31.316885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:31.316885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 07 03:47:31.316885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:31.316885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 215/429] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:31.320109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ddc311-3026-4f2a-9a4e-0af18a69057a None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:31.320109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ddc311-3026-4f2a-9a4e-0af18a69057a None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:31.320381 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ddc311-3026-4f2a-9a4e-0af18a69057a None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:31.325493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 216/430] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:31 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:31.327740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ed71a7-afbe-49c9-a64c-2683254e0f8e None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:31.328056 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ed71a7-afbe-49c9-a64c-2683254e0f8e None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:31.328392 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ed71a7-afbe-49c9-a64c-2683254e0f8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:31.351835 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-91ed71a7-afbe-49c9-a64c-2683254e0f8e None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:31.353930 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-91ed71a7-afbe-49c9-a64c-2683254e0f8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:31.398494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f05f914-c7d6-4db6-bed7-2abeb0eea435 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:31.428331 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:31.446157 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:31.451851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 217/431] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:31.452905 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ed71a7-afbe-49c9-a64c-2683254e0f8e None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:31.462306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcce2b6-312e-40e5-b7b6-f50139e05ffe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:31.463251 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcce2b6-312e-40e5-b7b6-f50139e05ffe None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:31.463764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcce2b6-312e-40e5-b7b6-f50139e05ffe None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:31.464254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcce2b6-312e-40e5-b7b6-f50139e05ffe None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:31.465123 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dcce2b6-312e-40e5-b7b6-f50139e05ffe None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:31.465429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dcce2b6-312e-40e5-b7b6-f50139e05ffe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:31.470168 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dcce2b6-312e-40e5-b7b6-f50139e05ffe None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:31.477994 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 215/432] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:31.484422 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcce2b6-312e-40e5-b7b6-f50139e05ffe None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:31.570613 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ed71a7-afbe-49c9-a64c-2683254e0f8e None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:31.616939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890ab6a5-a27b-414d-bd80-4b4ae61e648f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:31.617672 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890ab6a5-a27b-414d-bd80-4b4ae61e648f None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:31.617956 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890ab6a5-a27b-414d-bd80-4b4ae61e648f None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:31.618273 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890ab6a5-a27b-414d-bd80-4b4ae61e648f None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:31.618976 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890ab6a5-a27b-414d-bd80-4b4ae61e648f None admin] RBAC: Adding query filter params (name=member) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:31.619280 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890ab6a5-a27b-414d-bd80-4b4ae61e648f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:31.620725 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890ab6a5-a27b-414d-bd80-4b4ae61e648f None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:31.631748 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcce2b6-312e-40e5-b7b6-f50139e05ffe None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:31.632760 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 218/433] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:47:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:31.640922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890ab6a5-a27b-414d-bd80-4b4ae61e648f None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:31.691621 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2728c09-3e81-428a-818e-4ebcda0449eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:31.692404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2728c09-3e81-428a-818e-4ebcda0449eb None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:31.692788 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2728c09-3e81-428a-818e-4ebcda0449eb None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:31.693167 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2728c09-3e81-428a-818e-4ebcda0449eb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:31.719167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:31.761508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:31.784210 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890ab6a5-a27b-414d-bd80-4b4ae61e648f None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:31.792306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2728c09-3e81-428a-818e-4ebcda0449eb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:31.793791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2728c09-3e81-428a-818e-4ebcda0449eb None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:31.827338 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc446e1-4636-4e96-a4bb-e39baa0627a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:31.827956 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc446e1-4636-4e96-a4bb-e39baa0627a4 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:31.828232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc446e1-4636-4e96-a4bb-e39baa0627a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:31.828766 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc446e1-4636-4e96-a4bb-e39baa0627a4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:31.906049 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc446e1-4636-4e96-a4bb-e39baa0627a4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:31.907357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 216/434] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:31.908096 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc446e1-4636-4e96-a4bb-e39baa0627a4 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:31.908521 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2ff76d-5f5e-463f-9bd6-ea8cb760746f None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:31.908803 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2ff76d-5f5e-463f-9bd6-ea8cb760746f None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:31.909079 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2ff76d-5f5e-463f-9bd6-ea8cb760746f None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:31.910370 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 217/435] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:31.911071 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2ff76d-5f5e-463f-9bd6-ea8cb760746f None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:31.971660 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:32.003710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:32.017067 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 219/436] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:32.017910 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc446e1-4636-4e96-a4bb-e39baa0627a4 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:32.027484 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2ff76d-5f5e-463f-9bd6-ea8cb760746f None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:32.066347 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9f2fd2-86eb-4da3-9c3b-6c580d3f3a1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:32.067401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9f2fd2-86eb-4da3-9c3b-6c580d3f3a1c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:32.067886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9f2fd2-86eb-4da3-9c3b-6c580d3f3a1c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:32.068377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9f2fd2-86eb-4da3-9c3b-6c580d3f3a1c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:32.083083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a9f2fd2-86eb-4da3-9c3b-6c580d3f3a1c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:32.087885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-2a9f2fd2-86eb-4da3-9c3b-6c580d3f3a1c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:32.087885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:32.087885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:32.087885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:32.087885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:32.087885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:32.087885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:32.087885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:32.087885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:32.087885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:32.087885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:32.087885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:32.087885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:32.087885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:32.087885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:32.087885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:32.088972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:32.088972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:32.088972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:32.088972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:32.088972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:32.088972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:32.088972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:32.088972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:32.088972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:32.088972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:32.088972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:32.088972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:32.088972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:32.088972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:32.088972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:32.088972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:32.089757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:32.089757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:32.089757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:32.089757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:32.089757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:32.089757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:32.089757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:32.089757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:32.089757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:32.089757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:32.089757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:32.089757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:32.089757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 218/437] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:47:31 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:32.090499 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d977bf-8f2a-46c1-af07-0e12b274063c None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:32.090499 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d977bf-8f2a-46c1-af07-0e12b274063c None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:32.090657 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d977bf-8f2a-46c1-af07-0e12b274063c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:32.111620 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-f2d977bf-8f2a-46c1-af07-0e12b274063c None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:32.112828 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-f2d977bf-8f2a-46c1-af07-0e12b274063c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:32.126380 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc446e1-4636-4e96-a4bb-e39baa0627a4 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:32.160240 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b80cc7c-18c7-4205-93c5-f6e989bef66e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:32.160862 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b80cc7c-18c7-4205-93c5-f6e989bef66e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:32.161138 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b80cc7c-18c7-4205-93c5-f6e989bef66e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:32.161408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b80cc7c-18c7-4205-93c5-f6e989bef66e None admin] PATH_INFO: `/v3/projects/d6715600d626423b8e8a06d45bb53d08` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:32.162582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b80cc7c-18c7-4205-93c5-f6e989bef66e None admin] RBAC: Authorizing `identity:get_project(project_id=d6715600d626423b8e8a06d45bb53d08)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:32.164008 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b80cc7c-18c7-4205-93c5-f6e989bef66e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:32.165609 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 220/438] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:32 2022] GET /identity/v3/projects/d6715600d626423b8e8a06d45bb53d08 => generated 298 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:32.166708 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b80cc7c-18c7-4205-93c5-f6e989bef66e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:32.181226 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 219/439] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:32.182137 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d977bf-8f2a-46c1-af07-0e12b274063c None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:32.202363 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:32.261060 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:32.283645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b80cc7c-18c7-4205-93c5-f6e989bef66e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:32.299203 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d977bf-8f2a-46c1-af07-0e12b274063c None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:32.321295 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0cc347-53c9-4511-9154-6e2360bca529 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:32.322085 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0cc347-53c9-4511-9154-6e2360bca529 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:32.322396 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0cc347-53c9-4511-9154-6e2360bca529 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:32.323001 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0cc347-53c9-4511-9154-6e2360bca529 None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:32.324309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0cc347-53c9-4511-9154-6e2360bca529 None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:32.326278 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0cc347-53c9-4511-9154-6e2360bca529 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:32.332899 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 221/440] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:32 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:32.332899 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0cc347-53c9-4511-9154-6e2360bca529 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:32.336683 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef378f9-5f70-40a8-b82e-1fcec5303b5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:32.337247 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef378f9-5f70-40a8-b82e-1fcec5303b5f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:32.337530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef378f9-5f70-40a8-b82e-1fcec5303b5f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:32.337805 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef378f9-5f70-40a8-b82e-1fcec5303b5f None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:32.338508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef378f9-5f70-40a8-b82e-1fcec5303b5f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:32.338816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef378f9-5f70-40a8-b82e-1fcec5303b5f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:32.340192 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef378f9-5f70-40a8-b82e-1fcec5303b5f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:32.349789 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 220/441] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:32 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:32.355276 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef378f9-5f70-40a8-b82e-1fcec5303b5f None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:32.434874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:32.444192 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0cc347-53c9-4511-9154-6e2360bca529 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:32.466612 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef378f9-5f70-40a8-b82e-1fcec5303b5f None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:32.478619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dc5781-8806-476d-b627-cd5feb329cda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:32.479245 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dc5781-8806-476d-b627-cd5feb329cda None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:32.479523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dc5781-8806-476d-b627-cd5feb329cda None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:32.479801 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dc5781-8806-476d-b627-cd5feb329cda None admin] PATH_INFO: `/v3/groups/cae75895d7c9487494f7ba61ceeb3d10` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:32.480919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dc5781-8806-476d-b627-cd5feb329cda None admin] RBAC: Authorizing `identity:get_group(group_id=cae75895d7c9487494f7ba61ceeb3d10)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:32.482393 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dc5781-8806-476d-b627-cd5feb329cda None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:32.484603 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 222/442] 10.222.0.27 () {56 vars in 1180 bytes} [Wed Sep 7 03:47:32 2022] GET /identity/v3/groups/cae75895d7c9487494f7ba61ceeb3d10 => generated 226 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:32.485365 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dc5781-8806-476d-b627-cd5feb329cda None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:32.493813 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:32.502958 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ab3860-ebb7-41a6-b958-b4b08d5f85ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:32.503597 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ab3860-ebb7-41a6-b958-b4b08d5f85ef None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:32.503875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ab3860-ebb7-41a6-b958-b4b08d5f85ef None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:32.504149 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ab3860-ebb7-41a6-b958-b4b08d5f85ef None admin] PATH_INFO: `/v3/groups/cae75895d7c9487494f7ba61ceeb3d10` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:32.505283 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ab3860-ebb7-41a6-b958-b4b08d5f85ef None admin] RBAC: Authorizing `identity:get_group(group_id=cae75895d7c9487494f7ba61ceeb3d10)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:32.506772 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ab3860-ebb7-41a6-b958-b4b08d5f85ef None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:32.508484 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 221/443] 10.222.0.27 () {56 vars in 1180 bytes} [Wed Sep 7 03:47:32 2022] GET /identity/v3/groups/cae75895d7c9487494f7ba61ceeb3d10 => generated 226 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:32.509081 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ab3860-ebb7-41a6-b958-b4b08d5f85ef None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:32.588647 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dc5781-8806-476d-b627-cd5feb329cda None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:32.611766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ab3860-ebb7-41a6-b958-b4b08d5f85ef None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:32.622815 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c18f92a-0bcc-4820-aad5-6869d0bc43dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:32.623435 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c18f92a-0bcc-4820-aad5-6869d0bc43dd None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:32.624315 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c18f92a-0bcc-4820-aad5-6869d0bc43dd None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:32.624603 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c18f92a-0bcc-4820-aad5-6869d0bc43dd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:32.644561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4790e3-ceac-424b-96e3-0f22b4acbfba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:32.645192 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4790e3-ceac-424b-96e3-0f22b4acbfba None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:32.645474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4790e3-ceac-424b-96e3-0f22b4acbfba None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:32.645752 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4790e3-ceac-424b-96e3-0f22b4acbfba None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:32.654744 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:32.656294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4790e3-ceac-424b-96e3-0f22b4acbfba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:32.661764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-8f4790e3-ceac-424b-96e3-0f22b4acbfba None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:47:32.661764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:32.661764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:32.661764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:32.661764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:32.661764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:32.661764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:32.661764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:32.661764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:32.661764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:32.661764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:32.661764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:32.661764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:32.661764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:32.661764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:32.661764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:32.662783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:32.662783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:32.662783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:32.662783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:32.662783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:32.662783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:32.662783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:32.662783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:32.662783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:32.662783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:32.662783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:32.662783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:32.662783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:32.662783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:32.662783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:32.662783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:32.663587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:32.663587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:32.663587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:32.663587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:32.663587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:32.663587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:32.663587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:32.663587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:32.663587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:32.663587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:32.663587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:47:32.663587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:32.663587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 222/444] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:47:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:32.664245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4790e3-ceac-424b-96e3-0f22b4acbfba None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:32.693115 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c18f92a-0bcc-4820-aad5-6869d0bc43dd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:32.694619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c18f92a-0bcc-4820-aad5-6869d0bc43dd None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:32.720509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:32.765008 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4790e3-ceac-424b-96e3-0f22b4acbfba None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:32.790468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 223/445] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:32.791299 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c18f92a-0bcc-4820-aad5-6869d0bc43dd None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:32.798872 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759dec37-2be0-42c8-98c2-04a6b3e4bae5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:32.799489 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759dec37-2be0-42c8-98c2-04a6b3e4bae5 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:32.799771 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759dec37-2be0-42c8-98c2-04a6b3e4bae5 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:32.800045 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759dec37-2be0-42c8-98c2-04a6b3e4bae5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:32.800775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759dec37-2be0-42c8-98c2-04a6b3e4bae5 None admin] RBAC: Adding query filter params (group.id=cae75895d7c9487494f7ba61ceeb3d10, role.id=285107fb9b2449fe9f1d29f3075a246f, scope.project.id=d6715600d626423b8e8a06d45bb53d08) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:32.801083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759dec37-2be0-42c8-98c2-04a6b3e4bae5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:32.802571 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759dec37-2be0-42c8-98c2-04a6b3e4bae5 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:32.808362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 223/446] 10.222.0.27 () {56 vars in 1376 bytes} [Wed Sep 7 03:47:32 2022] GET /identity/v3/role_assignments?group.id=cae75895d7c9487494f7ba61ceeb3d10&scope.project.id=d6715600d626423b8e8a06d45bb53d08&role.id=285107fb9b2449fe9f1d29f3075a246f => generated 611 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:32.809134 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759dec37-2be0-42c8-98c2-04a6b3e4bae5 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:32.862840 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:32.903725 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c18f92a-0bcc-4820-aad5-6869d0bc43dd None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:32.932550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759dec37-2be0-42c8-98c2-04a6b3e4bae5 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:32.944277 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b75b87-eba9-408f-a008-279d765cad1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:32.944892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b75b87-eba9-408f-a008-279d765cad1b None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:32.945164 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b75b87-eba9-408f-a008-279d765cad1b None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:32.945431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b75b87-eba9-408f-a008-279d765cad1b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:32.946177 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b75b87-eba9-408f-a008-279d765cad1b None admin] RBAC: Adding query filter params (group.id=cae75895d7c9487494f7ba61ceeb3d10, role.id=34235948f8f943a99328516c2171cfff, scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:32.946485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b75b87-eba9-408f-a008-279d765cad1b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:32.947914 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b75b87-eba9-408f-a008-279d765cad1b None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:32.955616 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:32.955932 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 224/447] 10.222.0.27 () {56 vars in 1376 bytes} [Wed Sep 7 03:47:32 2022] GET /identity/v3/role_assignments?group.id=cae75895d7c9487494f7ba61ceeb3d10&scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54&role.id=34235948f8f943a99328516c2171cfff => generated 611 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:32.956793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b75b87-eba9-408f-a008-279d765cad1b None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:32.995985 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d36033-d604-4617-bdb8-34f0a825d483 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:32.996625 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d36033-d604-4617-bdb8-34f0a825d483 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:32.996910 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d36033-d604-4617-bdb8-34f0a825d483 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:32.997189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d36033-d604-4617-bdb8-34f0a825d483 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:32.997939 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d36033-d604-4617-bdb8-34f0a825d483 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:32.999608 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d36033-d604-4617-bdb8-34f0a825d483 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:33.001180 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d36033-d604-4617-bdb8-34f0a825d483 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:33.017032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 224/448] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:33.023887 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d36033-d604-4617-bdb8-34f0a825d483 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:33.094341 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b75b87-eba9-408f-a008-279d765cad1b None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:33.128961 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:33.140670 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d615d9ba-4977-49c3-bde0-8b4e6a7bc52b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:33.141613 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d615d9ba-4977-49c3-bde0-8b4e6a7bc52b None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:33.141979 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d615d9ba-4977-49c3-bde0-8b4e6a7bc52b None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:33.143144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d615d9ba-4977-49c3-bde0-8b4e6a7bc52b None admin] PATH_INFO: `/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:33.144320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d615d9ba-4977-49c3-bde0-8b4e6a7bc52b None admin] RBAC: Authorizing `identity:get_project(project_id=72fb26d349cf4298b0ad385cd5e2ee54)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:33.146055 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d615d9ba-4977-49c3-bde0-8b4e6a7bc52b None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:33.148064 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 225/449] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:32 2022] GET /identity/v3/projects/72fb26d349cf4298b0ad385cd5e2ee54 => generated 302 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:33.153480 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d615d9ba-4977-49c3-bde0-8b4e6a7bc52b None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:33.168998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d36033-d604-4617-bdb8-34f0a825d483 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:33.216170 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cad962-1d88-4d45-98a1-dc180450faa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:33.218245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cad962-1d88-4d45-98a1-dc180450faa3 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:33.218730 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cad962-1d88-4d45-98a1-dc180450faa3 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:33.219175 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cad962-1d88-4d45-98a1-dc180450faa3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:33.284042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:33.287354 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d615d9ba-4977-49c3-bde0-8b4e6a7bc52b None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:33.307933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cad962-1d88-4d45-98a1-dc180450faa3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:33.309412 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cad962-1d88-4d45-98a1-dc180450faa3 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:33.327606 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ccd64c-2b15-48b4-a0c8-0bf4c3fc6335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:33.328244 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ccd64c-2b15-48b4-a0c8-0bf4c3fc6335 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:33.328779 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ccd64c-2b15-48b4-a0c8-0bf4c3fc6335 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:33.329054 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ccd64c-2b15-48b4-a0c8-0bf4c3fc6335 None admin] PATH_INFO: `/v3/groups/cae75895d7c9487494f7ba61ceeb3d10` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:33.330229 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ccd64c-2b15-48b4-a0c8-0bf4c3fc6335 None admin] RBAC: Authorizing `identity:get_group(group_id=cae75895d7c9487494f7ba61ceeb3d10)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:33.331688 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ccd64c-2b15-48b4-a0c8-0bf4c3fc6335 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:33.333283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 226/450] 10.222.0.27 () {56 vars in 1180 bytes} [Wed Sep 7 03:47:33 2022] GET /identity/v3/groups/cae75895d7c9487494f7ba61ceeb3d10 => generated 226 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:33.339198 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ccd64c-2b15-48b4-a0c8-0bf4c3fc6335 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:33.421070 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:33.432653 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 225/451] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 07 03:47:33.439288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cad962-1d88-4d45-98a1-dc180450faa3 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:33.461293 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ccd64c-2b15-48b4-a0c8-0bf4c3fc6335 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:33.494407 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aacc8f9-7229-44cd-84d7-c29073484495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:33.494999 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aacc8f9-7229-44cd-84d7-c29073484495 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:33.495276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aacc8f9-7229-44cd-84d7-c29073484495 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:33.495547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aacc8f9-7229-44cd-84d7-c29073484495 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:33.496265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aacc8f9-7229-44cd-84d7-c29073484495 None admin] RBAC: Adding query filter params (group.id=cae75895d7c9487494f7ba61ceeb3d10, role.id=285107fb9b2449fe9f1d29f3075a246f, scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:33.496826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aacc8f9-7229-44cd-84d7-c29073484495 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:33.497999 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aacc8f9-7229-44cd-84d7-c29073484495 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:33.503906 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 227/452] 10.222.0.27 () {56 vars in 1376 bytes} [Wed Sep 7 03:47:33 2022] GET /identity/v3/role_assignments?group.id=cae75895d7c9487494f7ba61ceeb3d10&scope.project.id=72fb26d349cf4298b0ad385cd5e2ee54&role.id=285107fb9b2449fe9f1d29f3075a246f => generated 611 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:33.525715 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:33.547524 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cad962-1d88-4d45-98a1-dc180450faa3 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:33.584894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c2f0b6-1d20-4127-b143-e9b0092efdba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:33.585544 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c2f0b6-1d20-4127-b143-e9b0092efdba None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:33.585820 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c2f0b6-1d20-4127-b143-e9b0092efdba None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:33.586091 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c2f0b6-1d20-4127-b143-e9b0092efdba None admin] PATH_INFO: `/v3/projects/388f90b795764a26a3cfac5c43c50a76` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:33.587281 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c2f0b6-1d20-4127-b143-e9b0092efdba None admin] RBAC: Authorizing `identity:get_project(project_id=388f90b795764a26a3cfac5c43c50a76)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:33.588744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c2f0b6-1d20-4127-b143-e9b0092efdba None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:33.590463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 226/453] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:47:33 2022] GET /identity/v3/projects/388f90b795764a26a3cfac5c43c50a76 => generated 344 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:33.597063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aacc8f9-7229-44cd-84d7-c29073484495 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:33.703003 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aacc8f9-7229-44cd-84d7-c29073484495 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:33.729268 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:33.738235 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b48d37f-4c37-48d7-93fe-e3329a56c342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:33.738863 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b48d37f-4c37-48d7-93fe-e3329a56c342 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:33.739140 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b48d37f-4c37-48d7-93fe-e3329a56c342 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:33.739412 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b48d37f-4c37-48d7-93fe-e3329a56c342 None admin] PATH_INFO: `/v3/groups/7c25c11e1fdd4d1481b40cb29666e410` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:33.740515 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b48d37f-4c37-48d7-93fe-e3329a56c342 None admin] RBAC: Authorizing `identity:get_group(group_id=7c25c11e1fdd4d1481b40cb29666e410)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:33.741939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b48d37f-4c37-48d7-93fe-e3329a56c342 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:33.743519 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 228/454] 10.222.0.27 () {56 vars in 1180 bytes} [Wed Sep 7 03:47:33 2022] GET /identity/v3/groups/7c25c11e1fdd4d1481b40cb29666e410 => generated 229 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:33.749571 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c2f0b6-1d20-4127-b143-e9b0092efdba None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:33.852033 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c2f0b6-1d20-4127-b143-e9b0092efdba None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:33.886526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7411ad30-7baf-4335-bcc1-4f7d28b683ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:33.887139 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7411ad30-7baf-4335-bcc1-4f7d28b683ed None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:33.887420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7411ad30-7baf-4335-bcc1-4f7d28b683ed None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:33.887695 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7411ad30-7baf-4335-bcc1-4f7d28b683ed None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:33.888414 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7411ad30-7baf-4335-bcc1-4f7d28b683ed None admin] RBAC: Adding query filter params (group.id=7c25c11e1fdd4d1481b40cb29666e410, role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=388f90b795764a26a3cfac5c43c50a76) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:33.888724 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7411ad30-7baf-4335-bcc1-4f7d28b683ed None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:33.890409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7411ad30-7baf-4335-bcc1-4f7d28b683ed None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:33.896027 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 227/455] 10.222.0.27 () {56 vars in 1376 bytes} [Wed Sep 7 03:47:33 2022] GET /identity/v3/role_assignments?group.id=7c25c11e1fdd4d1481b40cb29666e410&scope.project.id=388f90b795764a26a3cfac5c43c50a76&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 611 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:36.558010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04138813-bbfe-4e4f-90a1-2fadcb0024c6 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:36.558561 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04138813-bbfe-4e4f-90a1-2fadcb0024c6 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:36.558561 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04138813-bbfe-4e4f-90a1-2fadcb0024c6 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:36.560174 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 229/456] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:36.566342 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea57c8b9-c0cc-4695-9db3-5bb92fb56e13 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:36.566822 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea57c8b9-c0cc-4695-9db3-5bb92fb56e13 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:36.566935 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea57c8b9-c0cc-4695-9db3-5bb92fb56e13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:36.586132 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-ea57c8b9-c0cc-4695-9db3-5bb92fb56e13 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:36.591391 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-ea57c8b9-c0cc-4695-9db3-5bb92fb56e13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:36.602242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:36.684646 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 228/457] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:47:36.731468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c289d5e9-3fcd-4072-8fb4-7b3f044ffa55 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:36.731798 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c289d5e9-3fcd-4072-8fb4-7b3f044ffa55 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:36.732025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c289d5e9-3fcd-4072-8fb4-7b3f044ffa55 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:36.733973 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 230/458] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:36.739104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48029b53-e087-4541-8829-2f8ca690caea None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:36.739392 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48029b53-e087-4541-8829-2f8ca690caea None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:36.739667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48029b53-e087-4541-8829-2f8ca690caea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:36.764164 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-48029b53-e087-4541-8829-2f8ca690caea None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:36.765676 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-48029b53-e087-4541-8829-2f8ca690caea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:36.863012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 229/459] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:47:36.930747 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b605f3ca-0e1c-43f5-b085-d48bbf92deaf None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:36.931075 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b605f3ca-0e1c-43f5-b085-d48bbf92deaf None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:36.931285 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b605f3ca-0e1c-43f5-b085-d48bbf92deaf None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:36.932627 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 231/460] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:36.947960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85eb2644-1807-42fd-b0b9-fa0fe896f319 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:36.948246 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85eb2644-1807-42fd-b0b9-fa0fe896f319 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:36.948522 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85eb2644-1807-42fd-b0b9-fa0fe896f319 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:36.978355 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-85eb2644-1807-42fd-b0b9-fa0fe896f319 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:36.979541 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-85eb2644-1807-42fd-b0b9-fa0fe896f319 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:36.987894 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b605f3ca-0e1c-43f5-b085-d48bbf92deaf None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:37.094321 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 230/461] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:37.156983 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b605f3ca-0e1c-43f5-b085-d48bbf92deaf None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:37.207286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255da279-7f89-45ea-a413-1d574bce33d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:37.208144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255da279-7f89-45ea-a413-1d574bce33d4 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:37.208570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255da279-7f89-45ea-a413-1d574bce33d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:37.208989 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255da279-7f89-45ea-a413-1d574bce33d4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:37.216746 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85eb2644-1807-42fd-b0b9-fa0fe896f319 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:37.222526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:37.225854 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255da279-7f89-45ea-a413-1d574bce33d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:37.227492 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255da279-7f89-45ea-a413-1d574bce33d4 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:37.250303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-255da279-7f89-45ea-a413-1d574bce33d4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:37.250303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:37.250303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:37.250303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:37.250303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:37.250303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:37.250303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:37.250303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:37.250303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:37.250303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:37.250303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:37.250303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:37.250303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:37.250303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:37.250303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:37.250303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:37.251458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:37.252346 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:37.252346 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:37.252346 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:37.259836 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 232/462] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:37.260887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255da279-7f89-45ea-a413-1d574bce33d4 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:37.352703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:37.389237 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85eb2644-1807-42fd-b0b9-fa0fe896f319 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:37.424925 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255da279-7f89-45ea-a413-1d574bce33d4 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:37.465627 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072f3e94-0f7f-4d30-8b3d-236e8bb86253 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:37.466905 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072f3e94-0f7f-4d30-8b3d-236e8bb86253 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:37.468051 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072f3e94-0f7f-4d30-8b3d-236e8bb86253 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:37.468598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072f3e94-0f7f-4d30-8b3d-236e8bb86253 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:37.469613 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2ccccc-474d-4f14-ac0e-81ef915277a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:37.470385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2ccccc-474d-4f14-ac0e-81ef915277a4 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:37.470731 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2ccccc-474d-4f14-ac0e-81ef915277a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:37.471070 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2ccccc-474d-4f14-ac0e-81ef915277a4 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:37.471799 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2ccccc-474d-4f14-ac0e-81ef915277a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:37.472258 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2ccccc-474d-4f14-ac0e-81ef915277a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:37.473648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2ccccc-474d-4f14-ac0e-81ef915277a4 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:37.484873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072f3e94-0f7f-4d30-8b3d-236e8bb86253 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:37.486938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072f3e94-0f7f-4d30-8b3d-236e8bb86253 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:37.487686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 233/463] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:37.488667 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2ccccc-474d-4f14-ac0e-81ef915277a4 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:37.510439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-072f3e94-0f7f-4d30-8b3d-236e8bb86253 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:37.510439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:37.510439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:37.510439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:37.510439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:37.510439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:37.510439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:37.510439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:37.510439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:37.510439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:37.510439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:37.510439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:37.510439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:37.510439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:37.510439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:37.510439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:37.511404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:37.512257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:37.512257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:37.512257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:37.515542 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 231/464] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:37.516851 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072f3e94-0f7f-4d30-8b3d-236e8bb86253 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:37.589623 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:37.609926 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2ccccc-474d-4f14-ac0e-81ef915277a4 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:37.620589 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:37.640156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072f3e94-0f7f-4d30-8b3d-236e8bb86253 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:37.648635 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d3d627-a2d6-4d81-8131-6a8ec00cb1a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:37.649264 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d3d627-a2d6-4d81-8131-6a8ec00cb1a5 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:37.649546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d3d627-a2d6-4d81-8131-6a8ec00cb1a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:37.649821 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d3d627-a2d6-4d81-8131-6a8ec00cb1a5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:37.660895 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d3d627-a2d6-4d81-8131-6a8ec00cb1a5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:37.662370 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d3d627-a2d6-4d81-8131-6a8ec00cb1a5 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:37.677800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-b8d3d627-a2d6-4d81-8131-6a8ec00cb1a5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:37.677800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:37.677800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:37.677800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:37.677800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:37.677800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:37.677800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:37.677800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:37.677800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:37.677800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:37.677800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:37.677800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:37.677800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:37.677800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:37.677800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:37.677800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:37.679017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:37.679858 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:37.679858 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:37.679858 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:37.679858 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 234/465] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:37.680073 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d3d627-a2d6-4d81-8131-6a8ec00cb1a5 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:37.686663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a91dd3d-88a1-42fb-ba91-897011c63ccd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:37.687374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a91dd3d-88a1-42fb-ba91-897011c63ccd None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:37.687717 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a91dd3d-88a1-42fb-ba91-897011c63ccd None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:37.687995 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a91dd3d-88a1-42fb-ba91-897011c63ccd None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:37.688669 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a91dd3d-88a1-42fb-ba91-897011c63ccd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:37.690143 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a91dd3d-88a1-42fb-ba91-897011c63ccd None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:37.713120 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 232/466] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:47:37 2022] POST /identity/v3/users => generated 246 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:47:37.714893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058395b8-4ef3-47eb-8919-1d4ec308a4a5 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:37.715394 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058395b8-4ef3-47eb-8919-1d4ec308a4a5 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:37.715871 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058395b8-4ef3-47eb-8919-1d4ec308a4a5 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:37.718054 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 233/467] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:37.719172 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058395b8-4ef3-47eb-8919-1d4ec308a4a5 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:37.817939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d3d627-a2d6-4d81-8131-6a8ec00cb1a5 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:37.861008 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a6723d-d0b7-4c25-8483-84bcc5b04986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:37.861731 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a6723d-d0b7-4c25-8483-84bcc5b04986 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:37.862067 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a6723d-d0b7-4c25-8483-84bcc5b04986 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:37.862456 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a6723d-d0b7-4c25-8483-84bcc5b04986 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:37.863166 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a6723d-d0b7-4c25-8483-84bcc5b04986 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:37.863664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a6723d-d0b7-4c25-8483-84bcc5b04986 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:37.864262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058395b8-4ef3-47eb-8919-1d4ec308a4a5 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:37.865206 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a6723d-d0b7-4c25-8483-84bcc5b04986 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:37.883336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 235/468] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:37.884434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:37.886815 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39822f71-93ed-4d21-8731-5d85dadfabbf None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:37.887260 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39822f71-93ed-4d21-8731-5d85dadfabbf None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:37.887642 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39822f71-93ed-4d21-8731-5d85dadfabbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:37.915900 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-39822f71-93ed-4d21-8731-5d85dadfabbf None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:37.917110 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-39822f71-93ed-4d21-8731-5d85dadfabbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:37.917586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9ac8d4-e914-4399-974b-56f01894b2e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:37.918248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9ac8d4-e914-4399-974b-56f01894b2e2 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:37.918536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9ac8d4-e914-4399-974b-56f01894b2e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:37.918812 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9ac8d4-e914-4399-974b-56f01894b2e2 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:37.919783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9ac8d4-e914-4399-974b-56f01894b2e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:37.921604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9ac8d4-e914-4399-974b-56f01894b2e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:37.923065 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9ac8d4-e914-4399-974b-56f01894b2e2 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:37.935686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:37.945341 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 234/469] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:37.946320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9ac8d4-e914-4399-974b-56f01894b2e2 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:38.005449 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 236/470] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:38.006692 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39822f71-93ed-4d21-8731-5d85dadfabbf None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:38.064764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9ac8d4-e914-4399-974b-56f01894b2e2 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:38.110905 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986e572b-7796-4be6-9c2d-29c90b5734d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:38.111763 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986e572b-7796-4be6-9c2d-29c90b5734d6 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:38.112464 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986e572b-7796-4be6-9c2d-29c90b5734d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:38.112746 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986e572b-7796-4be6-9c2d-29c90b5734d6 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:38.113454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986e572b-7796-4be6-9c2d-29c90b5734d6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:38.114954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986e572b-7796-4be6-9c2d-29c90b5734d6 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:38.129662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39822f71-93ed-4d21-8731-5d85dadfabbf None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:38.145646 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 235/471] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:47:37 2022] POST /identity/v3/users => generated 248 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:47:38.147616 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16755f7-3b15-4bb1-9af0-8ae137c9ba33 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:38.148124 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16755f7-3b15-4bb1-9af0-8ae137c9ba33 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:38.148456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16755f7-3b15-4bb1-9af0-8ae137c9ba33 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:38.150869 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 236/472] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:38 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:38.157413 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee41f06-2b2e-41fd-88ee-df2803f1e0d4 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:38.157697 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee41f06-2b2e-41fd-88ee-df2803f1e0d4 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:38.157973 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee41f06-2b2e-41fd-88ee-df2803f1e0d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:38.172324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caff121f-515f-4db8-a3e3-8c9bde6aa47a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:38.173155 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caff121f-515f-4db8-a3e3-8c9bde6aa47a None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:38.173544 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caff121f-515f-4db8-a3e3-8c9bde6aa47a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:38.173916 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caff121f-515f-4db8-a3e3-8c9bde6aa47a None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:38.174812 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caff121f-515f-4db8-a3e3-8c9bde6aa47a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:38.176430 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caff121f-515f-4db8-a3e3-8c9bde6aa47a None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:38.181185 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-dee41f06-2b2e-41fd-88ee-df2803f1e0d4 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:38.182516 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-dee41f06-2b2e-41fd-88ee-df2803f1e0d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:38.203791 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 237/473] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:47:38 2022] POST /identity/v3/users => generated 248 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:47:38.213478 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:38.270152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 237/474] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:47:38.427380 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caff121f-515f-4db8-a3e3-8c9bde6aa47a None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:38.439169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:38.541604 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caff121f-515f-4db8-a3e3-8c9bde6aa47a None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:38.566057 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee41f06-2b2e-41fd-88ee-df2803f1e0d4 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:38.584687 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746a0ff4-0ee8-42ac-be83-e01fbab88f32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:38.585629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746a0ff4-0ee8-42ac-be83-e01fbab88f32 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:38.586148 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746a0ff4-0ee8-42ac-be83-e01fbab88f32 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:38.586600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746a0ff4-0ee8-42ac-be83-e01fbab88f32 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:38.598837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746a0ff4-0ee8-42ac-be83-e01fbab88f32 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:38.600310 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746a0ff4-0ee8-42ac-be83-e01fbab88f32 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:38.616826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-746a0ff4-0ee8-42ac-be83-e01fbab88f32 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:38.616826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:38.616826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:38.616826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:38.616826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:38.616826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:38.616826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:38.616826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:38.616826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:38.616826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:38.616826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:38.616826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:38.616826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:38.616826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:38.616826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:38.616826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:38.618226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:38.619093 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:38.619093 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:38.619093 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:38.620871 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 238/475] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:38.627228 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746a0ff4-0ee8-42ac-be83-e01fbab88f32 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:38.690108 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee41f06-2b2e-41fd-88ee-df2803f1e0d4 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:38.713567 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:38.730081 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb33983b-8aa7-463e-81b5-805dbc3f36bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:38.730743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb33983b-8aa7-463e-81b5-805dbc3f36bb None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:38.731113 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb33983b-8aa7-463e-81b5-805dbc3f36bb None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:38.731289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb33983b-8aa7-463e-81b5-805dbc3f36bb None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:38.731988 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb33983b-8aa7-463e-81b5-805dbc3f36bb None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:38.734073 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb33983b-8aa7-463e-81b5-805dbc3f36bb None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:38.744846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.sql.core [None req-bb33983b-8aa7-463e-81b5-805dbc3f36bb None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 07 03:47:38.744846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 07 03:47:38.744846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [parameters: {'id': 'cc98f479ab84475fa72b5906f5d23b72', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 07 03:47:38.744846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81391) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 07 03:47:38.762954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-bb33983b-8aa7-463e-81b5-805dbc3f36bb None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 07 03:47:38.762954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:38.762954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 07 03:47:38.762954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 07 03:47:38.762954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 07 03:47:38.762954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 07 03:47:38.762954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 07 03:47:38.762954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application result = self._query(query) Sep 07 03:47:38.762954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 07 03:47:38.762954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application conn.query(q) Sep 07 03:47:38.762954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 07 03:47:38.762954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 07 03:47:38.762954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 07 03:47:38.762954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application result.read() Sep 07 03:47:38.762954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return ref.to_dict() Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 07 03:47:38.764450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application next(self.gen) Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application yield resource Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application next(self.gen) Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self.session.rollback() Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self.force_reraise() Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise self.value Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application session.commit() Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 07 03:47:38.765386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._prepare_impl() Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self.session.flush() Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._flush(objects) Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application compat.raise_( Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application flush_context.execute() Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rec.execute(self) Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 07 03:47:38.766919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 07 03:47:38.767810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 07 03:47:38.767810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 07 03:47:38.767810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 07 03:47:38.767810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 07 03:47:38.767810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 07 03:47:38.767810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 07 03:47:38.767810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 07 03:47:38.767810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 07 03:47:38.767810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 07 03:47:38.767810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 07 03:47:38.767810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 07 03:47:38.767810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 07 03:47:38.767810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 07 03:47:38.767810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 07 03:47:38.767810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 07 03:47:38.767810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception Sep 07 03:47:38.768604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 07 03:47:38.768604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 07 03:47:38.768604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 07 03:47:38.768604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 07 03:47:38.768604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 07 03:47:38.768604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application result = self._query(query) Sep 07 03:47:38.768604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 07 03:47:38.768604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application conn.query(q) Sep 07 03:47:38.768604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 07 03:47:38.768604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 07 03:47:38.768604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 07 03:47:38.768604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application result.read() Sep 07 03:47:38.768604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 07 03:47:38.768604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 07 03:47:38.768604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 07 03:47:38.768604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 07 03:47:38.769396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 07 03:47:38.769396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 07 03:47:38.769396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 07 03:47:38.769396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 07 03:47:38.769396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 07 03:47:38.769396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 07 03:47:38.769396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [parameters: {'id': 'cc98f479ab84475fa72b5906f5d23b72', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 07 03:47:38.769396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 07 03:47:38.769396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:38.769396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:47:38.769396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:38.769396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:38.769396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:38.769396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:38.769396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:38.769396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:38.771466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:38.771466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:38.771466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:38.771466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:38.771466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:38.771466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:38.771466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 07 03:47:38.771466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 07 03:47:38.771466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:38.771466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:38.771466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 07 03:47:38.771466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 07 03:47:38.771466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 07 03:47:38.771466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 07 03:47:38.771466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 07 03:47:38.771466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:38.773553 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 238/476] 10.222.0.27 () {60 vars in 1131 bytes} [Wed Sep 7 03:47:38 2022] POST /identity/v3/roles => generated 144 bytes in 199 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:38.773553 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb33983b-8aa7-463e-81b5-805dbc3f36bb None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:38.780349 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746a0ff4-0ee8-42ac-be83-e01fbab88f32 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:38.800075 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:38.829227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340e97d6-e8df-4f42-851c-c0c5aeda1099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:38.830571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340e97d6-e8df-4f42-851c-c0c5aeda1099 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:38.831105 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340e97d6-e8df-4f42-851c-c0c5aeda1099 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:38.831531 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340e97d6-e8df-4f42-851c-c0c5aeda1099 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:38.832346 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340e97d6-e8df-4f42-851c-c0c5aeda1099 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:38.832962 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340e97d6-e8df-4f42-851c-c0c5aeda1099 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:38.835204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340e97d6-e8df-4f42-851c-c0c5aeda1099 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:38.852642 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 239/477] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:38.858420 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340e97d6-e8df-4f42-851c-c0c5aeda1099 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:38.900826 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb33983b-8aa7-463e-81b5-805dbc3f36bb None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:38.947244 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfb654f-904d-4d8a-8b41-7cb84ba078b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:38.948138 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfb654f-904d-4d8a-8b41-7cb84ba078b1 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:38.948567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfb654f-904d-4d8a-8b41-7cb84ba078b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:38.948987 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfb654f-904d-4d8a-8b41-7cb84ba078b1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:38.973234 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfb654f-904d-4d8a-8b41-7cb84ba078b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:38.978567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-7dfb654f-904d-4d8a-8b41-7cb84ba078b1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:38.978567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:38.978567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:38.978567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:38.978567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:38.978567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:38.978567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:38.978567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:38.978567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:38.978567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:38.978567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:38.978567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:38.978567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:38.978567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:38.978567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:38.978567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:38.979631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:38.979631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:38.979631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:38.979631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:38.979631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:38.979631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:38.979631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:38.979631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:38.979631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:38.979631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:38.979631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:38.979631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:38.979631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:38.979631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:38.979631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:38.979631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:38.980558 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:38.980558 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:38.980558 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:38.980558 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:38.980558 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:38.980558 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:38.980558 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:38.980558 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:38.980558 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:38.980558 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:38.980558 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 07 03:47:38.980558 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:38.988516 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 239/478] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:47:38 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:38.995449 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfb654f-904d-4d8a-8b41-7cb84ba078b1 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:39.018934 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340e97d6-e8df-4f42-851c-c0c5aeda1099 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:39.055450 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:39.083556 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fda39b-9bfa-47bc-99d2-7cae4c73987d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:39.084171 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fda39b-9bfa-47bc-99d2-7cae4c73987d None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:39.084451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fda39b-9bfa-47bc-99d2-7cae4c73987d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:39.084722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fda39b-9bfa-47bc-99d2-7cae4c73987d None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:39.085376 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fda39b-9bfa-47bc-99d2-7cae4c73987d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:39.086850 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fda39b-9bfa-47bc-99d2-7cae4c73987d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:39.130821 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 240/479] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:47:38 2022] POST /identity/v3/users => generated 249 bytes in 273 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 07 03:47:39.136533 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:39.166978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfb654f-904d-4d8a-8b41-7cb84ba078b1 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:39.216761 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b94828-bc56-4ac0-954a-7f6f68d202b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:39.218307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b94828-bc56-4ac0-954a-7f6f68d202b7 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:39.218640 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b94828-bc56-4ac0-954a-7f6f68d202b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:39.218923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b94828-bc56-4ac0-954a-7f6f68d202b7 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:39.219677 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b94828-bc56-4ac0-954a-7f6f68d202b7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:39.219990 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b94828-bc56-4ac0-954a-7f6f68d202b7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:39.221496 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b94828-bc56-4ac0-954a-7f6f68d202b7 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:39.240556 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 240/480] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:38 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:40.449405 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1092984-c011-4042-bc16-7cc7e74d8b0a None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:40.449909 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1092984-c011-4042-bc16-7cc7e74d8b0a None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:40.449984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1092984-c011-4042-bc16-7cc7e74d8b0a None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:40.451784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 241/481] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:40.457831 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafa472a-d242-455b-b270-95886dabe255 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:40.458241 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafa472a-d242-455b-b270-95886dabe255 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:40.458424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafa472a-d242-455b-b270-95886dabe255 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:40.486917 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-aafa472a-d242-455b-b270-95886dabe255 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:40.488122 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-aafa472a-d242-455b-b270-95886dabe255 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:40.572767 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 241/482] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:47:41.033379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1092984-c011-4042-bc16-7cc7e74d8b0a None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:41.171718 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1092984-c011-4042-bc16-7cc7e74d8b0a None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:41.209637 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52b1e4c-c08c-462a-8944-0315dfdbbd2e None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:41.210022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52b1e4c-c08c-462a-8944-0315dfdbbd2e None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:41.210338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52b1e4c-c08c-462a-8944-0315dfdbbd2e None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:41.212380 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8731a2-9233-483f-866d-db6a67612832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:41.213019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8731a2-9233-483f-866d-db6a67612832 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:41.213245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 242/483] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:47:41.213345 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8731a2-9233-483f-866d-db6a67612832 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:41.213555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8731a2-9233-483f-866d-db6a67612832 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:41.218271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13edb3af-3be1-41cf-9304-5bb1f15b3a94 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:41.218654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13edb3af-3be1-41cf-9304-5bb1f15b3a94 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:41.218933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13edb3af-3be1-41cf-9304-5bb1f15b3a94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:41.237045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8731a2-9233-483f-866d-db6a67612832 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:41.242431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-8e8731a2-9233-483f-866d-db6a67612832 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:41.242431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:41.242431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:41.242431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:41.242431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:41.242431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:41.242431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:41.242431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:41.242431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:41.242431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:41.242431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:41.242431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:41.242431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:41.242431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:41.242431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:41.242431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:41.243316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:41.243316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:41.243316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:41.243316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:41.243316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:41.243316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:41.243316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:41.243316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:41.243316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:41.243316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:41.243316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:41.243316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:41.243316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:41.243316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:41.243316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:41.243316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:41.244097 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:41.244097 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:41.244097 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:41.244097 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:41.244097 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:41.244097 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:41.244097 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:41.244097 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:41.244097 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:41.244097 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:41.244097 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:41.244097 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:41.245196 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 242/484] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:41.249878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8731a2-9233-483f-866d-db6a67612832 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:41.259171 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-13edb3af-3be1-41cf-9304-5bb1f15b3a94 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:41.260373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-13edb3af-3be1-41cf-9304-5bb1f15b3a94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:41.291436 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:41.380609 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8731a2-9233-483f-866d-db6a67612832 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:41.395521 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 243/485] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:41.399200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6368ed78-4116-4d60-885e-4a9884d42374 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:41.399488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6368ed78-4116-4d60-885e-4a9884d42374 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:41.399764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6368ed78-4116-4d60-885e-4a9884d42374 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:41.401317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 244/486] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:41 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:41.411406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61862e20-54d9-42f7-a262-ae2d37fd8553 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:41.411686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61862e20-54d9-42f7-a262-ae2d37fd8553 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:41.412014 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61862e20-54d9-42f7-a262-ae2d37fd8553 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:41.426829 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:41.445455 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310721ef-1e53-40cf-91d1-3cbef8a7d0d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:41.446144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310721ef-1e53-40cf-91d1-3cbef8a7d0d9 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:41.446441 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310721ef-1e53-40cf-91d1-3cbef8a7d0d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:41.446766 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310721ef-1e53-40cf-91d1-3cbef8a7d0d9 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:41.447449 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310721ef-1e53-40cf-91d1-3cbef8a7d0d9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:41.447755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310721ef-1e53-40cf-91d1-3cbef8a7d0d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:41.449146 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310721ef-1e53-40cf-91d1-3cbef8a7d0d9 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:41.451481 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-61862e20-54d9-42f7-a262-ae2d37fd8553 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:41.452762 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-61862e20-54d9-42f7-a262-ae2d37fd8553 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:41.470230 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 243/487] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:47:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:41.476841 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310721ef-1e53-40cf-91d1-3cbef8a7d0d9 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:41.581772 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 245/488] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:47:41.634067 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:41.641073 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310721ef-1e53-40cf-91d1-3cbef8a7d0d9 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:41.678215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09955c9f-09d8-473e-9a68-6f7681d13894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:41.678879 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09955c9f-09d8-473e-9a68-6f7681d13894 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:41.679173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09955c9f-09d8-473e-9a68-6f7681d13894 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:41.679457 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09955c9f-09d8-473e-9a68-6f7681d13894 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:41.691553 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09955c9f-09d8-473e-9a68-6f7681d13894 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:41.693087 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09955c9f-09d8-473e-9a68-6f7681d13894 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:41.709694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-09955c9f-09d8-473e-9a68-6f7681d13894 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:41.709694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:41.709694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:41.709694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:41.709694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:41.709694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:41.709694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:41.709694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:41.709694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:41.709694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:41.709694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:41.709694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:41.709694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:41.709694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:41.709694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:41.709694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:41.710638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:41.711565 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:41.711565 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:41.711565 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:41.716248 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 244/489] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:41.719311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61862e20-54d9-42f7-a262-ae2d37fd8553 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:41.731142 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09955c9f-09d8-473e-9a68-6f7681d13894 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:41.854144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61862e20-54d9-42f7-a262-ae2d37fd8553 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:41.885051 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09955c9f-09d8-473e-9a68-6f7681d13894 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:41.891374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2442d54-1409-4669-9f4a-a759e155721e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:41.891988 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2442d54-1409-4669-9f4a-a759e155721e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:41.892261 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2442d54-1409-4669-9f4a-a759e155721e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:41.892531 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2442d54-1409-4669-9f4a-a759e155721e None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:41.893179 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2442d54-1409-4669-9f4a-a759e155721e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:41.893478 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2442d54-1409-4669-9f4a-a759e155721e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:41.894871 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2442d54-1409-4669-9f4a-a759e155721e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:41.896454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:41.908287 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 246/490] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:41.908287 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eadc627-78d1-460a-b765-d7579fe317dc None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:41.908287 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eadc627-78d1-460a-b765-d7579fe317dc None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:41.908287 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eadc627-78d1-460a-b765-d7579fe317dc None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:41.908287 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 247/491] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:41 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:41.908287 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eadc627-78d1-460a-b765-d7579fe317dc None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:41.928022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a619d35b-8d95-48f7-ac71-1a575145ef6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:41.928937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a619d35b-8d95-48f7-ac71-1a575145ef6d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:41.929225 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a619d35b-8d95-48f7-ac71-1a575145ef6d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:41.929501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a619d35b-8d95-48f7-ac71-1a575145ef6d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:41.940541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a619d35b-8d95-48f7-ac71-1a575145ef6d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:41.945834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-a619d35b-8d95-48f7-ac71-1a575145ef6d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:41.945834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:41.945834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:41.945834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:41.945834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:41.945834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:41.945834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:41.945834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:41.945834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:41.945834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:41.945834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:41.945834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:41.945834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:41.945834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:41.945834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:41.945834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:41.946768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:41.946768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:41.946768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:41.946768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:41.946768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:41.946768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:41.946768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:41.946768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:41.946768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:41.946768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:41.946768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:41.946768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:41.946768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:41.946768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:41.946768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:41.946768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:41.947583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:41.947583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:41.947583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:41.947583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:41.947583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:41.947583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:41.947583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:41.947583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:41.947583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:41.947583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:41.947583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:41.947583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:41.947583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 245/492] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:41.948242 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a619d35b-8d95-48f7-ac71-1a575145ef6d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:41.960020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:42.019470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eadc627-78d1-460a-b765-d7579fe317dc None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:42.050381 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a619d35b-8d95-48f7-ac71-1a575145ef6d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:42.052819 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa824ce-30c4-40eb-ad94-d29c548d97ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:42.053446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa824ce-30c4-40eb-ad94-d29c548d97ae None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:42.053726 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa824ce-30c4-40eb-ad94-d29c548d97ae None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:42.053997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa824ce-30c4-40eb-ad94-d29c548d97ae None admin] PATH_INFO: `/v3/roles/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:42.064203 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa824ce-30c4-40eb-ad94-d29c548d97ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:42.069508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-3fa824ce-30c4-40eb-ad94-d29c548d97ae None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:42.069508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:42.069508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:42.069508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:42.069508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:42.069508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:42.069508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:42.069508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:42.069508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:42.069508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:42.069508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:42.069508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:42.069508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:42.069508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:42.069508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:42.069508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:42.070442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:42.070442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:42.070442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:42.070442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:42.070442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:42.070442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:42.070442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:42.070442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:42.070442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:42.070442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:42.070442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:42.070442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:42.070442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:42.070442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:42.070442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:42.070442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:42.071247 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:42.071247 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:42.071247 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:42.071247 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:42.071247 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:42.071247 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:42.071247 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:42.071247 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:42.071247 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:42.071247 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:42.071247 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:42.071247 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:42.075555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 248/493] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:42.076820 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9f085c-9248-430c-b8ee-15de0ba465a3 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:42.077083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9f085c-9248-430c-b8ee-15de0ba465a3 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:42.077361 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9f085c-9248-430c-b8ee-15de0ba465a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:42.083840 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb380ca1-c214-4604-b849-fa7d979a326e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:42.084468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb380ca1-c214-4604-b849-fa7d979a326e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:42.084748 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb380ca1-c214-4604-b849-fa7d979a326e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:42.085026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb380ca1-c214-4604-b849-fa7d979a326e None admin] PATH_INFO: `/v3/users/nova` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:42.093121 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb380ca1-c214-4604-b849-fa7d979a326e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:42.094613 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb380ca1-c214-4604-b849-fa7d979a326e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:42.095213 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-dd9f085c-9248-430c-b8ee-15de0ba465a3 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:42.096399 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-dd9f085c-9248-430c-b8ee-15de0ba465a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:42.106988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-cb380ca1-c214-4604-b849-fa7d979a326e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:47:42.106988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:42.106988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:42.106988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:42.106988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:42.106988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:42.106988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:42.106988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:42.106988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:42.106988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:42.106988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:42.106988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:42.106988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:47:42.106988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:47:42.106988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:47:42.106988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:42.107990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:42.108889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:42.108889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:42.108889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:47:42.108889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:47:42.108889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:47:42.108889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:47:42.108889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:47:42.108889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:47:42.108889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:47:42.108889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:42.108889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 246/494] 10.222.0.27 () {56 vars in 1093 bytes} [Wed Sep 7 03:47:41 2022] GET /identity/v3/users/nova => generated 82 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:42.109492 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb380ca1-c214-4604-b849-fa7d979a326e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:42.149911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:42.167271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 249/495] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:42.168498 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f339abfd-fb52-4017-871a-6ad4db8ad3fc None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:42.168785 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f339abfd-fb52-4017-871a-6ad4db8ad3fc None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:42.169060 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f339abfd-fb52-4017-871a-6ad4db8ad3fc None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:42.170400 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 250/496] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:42.171085 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f339abfd-fb52-4017-871a-6ad4db8ad3fc None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:42.189557 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:42.225918 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb380ca1-c214-4604-b849-fa7d979a326e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:42.265425 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b4c9a9-b370-4f91-a309-bb84ff865de7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:42.266147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b4c9a9-b370-4f91-a309-bb84ff865de7 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:42.266502 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b4c9a9-b370-4f91-a309-bb84ff865de7 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:42.266980 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b4c9a9-b370-4f91-a309-bb84ff865de7 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:42.267634 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b4c9a9-b370-4f91-a309-bb84ff865de7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:42.268008 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b4c9a9-b370-4f91-a309-bb84ff865de7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:42.269386 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b4c9a9-b370-4f91-a309-bb84ff865de7 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:42.282160 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 247/497] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:47:42 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:42.282160 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b4c9a9-b370-4f91-a309-bb84ff865de7 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:42.284261 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f339abfd-fb52-4017-871a-6ad4db8ad3fc None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:42.321544 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50ac2b3-f980-4a96-a348-6e9d2b53d07d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:42.322178 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50ac2b3-f980-4a96-a348-6e9d2b53d07d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:42.322461 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50ac2b3-f980-4a96-a348-6e9d2b53d07d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:42.322731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50ac2b3-f980-4a96-a348-6e9d2b53d07d None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:42.323399 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50ac2b3-f980-4a96-a348-6e9d2b53d07d None admin] RBAC: Adding query filter params (name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:42.323703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50ac2b3-f980-4a96-a348-6e9d2b53d07d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:42.325092 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50ac2b3-f980-4a96-a348-6e9d2b53d07d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:42.334452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 251/498] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:47:42 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:42.335434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebaadbe-5b26-4985-a17c-715c9a08867c None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:42.335717 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebaadbe-5b26-4985-a17c-715c9a08867c None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:42.335990 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebaadbe-5b26-4985-a17c-715c9a08867c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:42.354156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-0ebaadbe-5b26-4985-a17c-715c9a08867c None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:42.355343 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-0ebaadbe-5b26-4985-a17c-715c9a08867c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:42.390259 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b4c9a9-b370-4f91-a309-bb84ff865de7 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:42.402040 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:42.420276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:42.426920 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5212e6-12c6-41d3-a52b-896e9415a33e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:42.427527 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5212e6-12c6-41d3-a52b-896e9415a33e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:42.427804 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5212e6-12c6-41d3-a52b-896e9415a33e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:42.428076 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5212e6-12c6-41d3-a52b-896e9415a33e None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:42.428832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5212e6-12c6-41d3-a52b-896e9415a33e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:42.429140 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5212e6-12c6-41d3-a52b-896e9415a33e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:42.430602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5212e6-12c6-41d3-a52b-896e9415a33e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:42.442421 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 252/499] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:42.442421 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebaadbe-5b26-4985-a17c-715c9a08867c None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:42.470549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 248/500] 10.222.0.27 () {56 vars in 1133 bytes} [Wed Sep 7 03:47:42 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:42.471463 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5212e6-12c6-41d3-a52b-896e9415a33e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:42.550212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebaadbe-5b26-4985-a17c-715c9a08867c None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:42.581129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5212e6-12c6-41d3-a52b-896e9415a33e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:42.585862 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0da074-edb1-414f-beb9-1cbd01984b43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:42.586583 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0da074-edb1-414f-beb9-1cbd01984b43 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:42.586861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0da074-edb1-414f-beb9-1cbd01984b43 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:42.587132 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0da074-edb1-414f-beb9-1cbd01984b43 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:42.598091 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0da074-edb1-414f-beb9-1cbd01984b43 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:42.599564 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0da074-edb1-414f-beb9-1cbd01984b43 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:42.613978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-ab0da074-edb1-414f-beb9-1cbd01984b43 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:42.613978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:42.613978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:42.613978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:42.613978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:42.613978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:42.613978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:42.613978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:42.613978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:42.613978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:42.613978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:42.613978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:42.613978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:42.613978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:42.613978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:42.613978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:42.615080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:42.616089 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:42.616089 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:42.616089 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:42.616089 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 253/501] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:42.616335 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0da074-edb1-414f-beb9-1cbd01984b43 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:42.618874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad15a11-ac85-42b1-868d-38763fc09f05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:42.619622 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad15a11-ac85-42b1-868d-38763fc09f05 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:42.619851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad15a11-ac85-42b1-868d-38763fc09f05 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:42.620047 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad15a11-ac85-42b1-868d-38763fc09f05 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:42.630915 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad15a11-ac85-42b1-868d-38763fc09f05 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:42.632358 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad15a11-ac85-42b1-868d-38763fc09f05 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:42.647029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-9ad15a11-ac85-42b1-868d-38763fc09f05 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:42.647029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:42.647029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:42.647029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:42.647029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:42.647029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:42.647029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:42.647029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:42.647029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:42.647029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:42.647029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:42.647029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:42.647029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:42.647029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:42.647029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:42.647029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:42.647883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:42.648724 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:42.648724 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:42.648724 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:42.648724 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 249/502] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:42.648935 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad15a11-ac85-42b1-868d-38763fc09f05 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:42.653355 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:42.694135 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:42.730655 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0da074-edb1-414f-beb9-1cbd01984b43 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:42.756779 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad15a11-ac85-42b1-868d-38763fc09f05 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:42.765305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a52e9f-f484-4393-9d4b-05eb7c864d96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:42.765935 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a52e9f-f484-4393-9d4b-05eb7c864d96 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:42.766254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a52e9f-f484-4393-9d4b-05eb7c864d96 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:42.766542 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a52e9f-f484-4393-9d4b-05eb7c864d96 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:42.776875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a52e9f-f484-4393-9d4b-05eb7c864d96 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:42.778363 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a52e9f-f484-4393-9d4b-05eb7c864d96 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:42.789902 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdca2aef-b12f-430f-a26f-3f04f3c1b95c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:42.790541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdca2aef-b12f-430f-a26f-3f04f3c1b95c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:42.790820 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdca2aef-b12f-430f-a26f-3f04f3c1b95c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:42.791094 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdca2aef-b12f-430f-a26f-3f04f3c1b95c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:42.792674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-c0a52e9f-f484-4393-9d4b-05eb7c864d96 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:42.792674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:42.792674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:42.792674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:42.792674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:42.792674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:42.792674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:42.792674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:42.792674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:42.792674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:42.792674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:42.792674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:42.792674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:42.792674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:42.792674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:42.792674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:42.793663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:42.794569 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:42.794569 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:42.794569 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:42.794569 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 254/503] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:42.794569 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a52e9f-f484-4393-9d4b-05eb7c864d96 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:42.859674 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdca2aef-b12f-430f-a26f-3f04f3c1b95c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:42.861146 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdca2aef-b12f-430f-a26f-3f04f3c1b95c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:42.876550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:42.895725 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a52e9f-f484-4393-9d4b-05eb7c864d96 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:42.902235 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:42.928631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6eb0a7-130a-4c0c-8bfb-a3436def1972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:42.929237 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6eb0a7-130a-4c0c-8bfb-a3436def1972 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:42.929518 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6eb0a7-130a-4c0c-8bfb-a3436def1972 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:42.929795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6eb0a7-130a-4c0c-8bfb-a3436def1972 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:42.930481 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6eb0a7-130a-4c0c-8bfb-a3436def1972 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:42.930789 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6eb0a7-130a-4c0c-8bfb-a3436def1972 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:42.932173 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6eb0a7-130a-4c0c-8bfb-a3436def1972 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:42.944197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 255/504] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:42.944997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6eb0a7-130a-4c0c-8bfb-a3436def1972 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:42.963510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 250/505] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:42.964365 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdca2aef-b12f-430f-a26f-3f04f3c1b95c None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:43.047244 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6eb0a7-130a-4c0c-8bfb-a3436def1972 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:43.061324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdca2aef-b12f-430f-a26f-3f04f3c1b95c None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:43.081763 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbee395f-ff48-4adf-b616-08b039cdd2b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:43.082400 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbee395f-ff48-4adf-b616-08b039cdd2b6 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:43.082682 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbee395f-ff48-4adf-b616-08b039cdd2b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:43.082956 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbee395f-ff48-4adf-b616-08b039cdd2b6 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:43.083603 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbee395f-ff48-4adf-b616-08b039cdd2b6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:43.083905 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbee395f-ff48-4adf-b616-08b039cdd2b6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:43.085279 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbee395f-ff48-4adf-b616-08b039cdd2b6 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:43.094138 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:43.096074 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05ecafd-2ecf-4394-aba4-580d35c40a3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:43.096700 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05ecafd-2ecf-4394-aba4-580d35c40a3f None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:43.101303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05ecafd-2ecf-4394-aba4-580d35c40a3f None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:43.101438 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 256/506] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:43.101546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05ecafd-2ecf-4394-aba4-580d35c40a3f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:43.101632 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbee395f-ff48-4adf-b616-08b039cdd2b6 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:43.105279 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:43.109225 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05ecafd-2ecf-4394-aba4-580d35c40a3f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:43.114109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-a05ecafd-2ecf-4394-aba4-580d35c40a3f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:43.114109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:43.114109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:43.114109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:43.114109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:43.114109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:43.114109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:43.114109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:43.114109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:43.114109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:43.114109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:43.114109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:43.114109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:43.114109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:43.114109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:43.114109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:43.115151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:43.115151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:43.115151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:43.115151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:43.115151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:43.115151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:43.115151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:43.115151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:43.115151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:43.115151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:43.115151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:43.115151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:43.115151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:43.115151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:43.115151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:43.115151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:43.115958 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:43.115958 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:43.115958 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:43.115958 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:43.115958 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:43.115958 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:43.115958 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:43.115958 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:43.115958 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:43.115958 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:43.115958 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:43.115958 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:43.115958 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 251/507] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:42 2022] GET /identity/v3/roles/service => generated 85 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:43.116601 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05ecafd-2ecf-4394-aba4-580d35c40a3f None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:43.198970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbee395f-ff48-4adf-b616-08b039cdd2b6 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:43.215512 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05ecafd-2ecf-4394-aba4-580d35c40a3f None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:43.233029 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71662aae-eba4-4808-8475-453a29a8ae1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:43.233632 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71662aae-eba4-4808-8475-453a29a8ae1d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:43.233914 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71662aae-eba4-4808-8475-453a29a8ae1d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:43.234219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71662aae-eba4-4808-8475-453a29a8ae1d None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:43.234862 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71662aae-eba4-4808-8475-453a29a8ae1d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:43.235163 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71662aae-eba4-4808-8475-453a29a8ae1d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:43.236528 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71662aae-eba4-4808-8475-453a29a8ae1d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:43.247996 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 257/508] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:43.248838 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71662aae-eba4-4808-8475-453a29a8ae1d None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:43.253487 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3aedc36-7471-474a-8b87-e336e33b444e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:43.254137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3aedc36-7471-474a-8b87-e336e33b444e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:43.254428 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3aedc36-7471-474a-8b87-e336e33b444e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:43.254703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3aedc36-7471-474a-8b87-e336e33b444e None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:43.265022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3aedc36-7471-474a-8b87-e336e33b444e None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:43.266517 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3aedc36-7471-474a-8b87-e336e33b444e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:43.278139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-b3aedc36-7471-474a-8b87-e336e33b444e None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 07 03:47:43.278139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:43.278139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:43.278139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:43.278139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:43.278139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:43.278139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:43.278139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:43.278139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:43.278139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:43.278139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:43.278139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:43.278139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:47:43.278139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:47:43.278139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:47:43.278139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:43.279223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:43.281027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:43.281027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:43.281027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:47:43.281027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:47:43.281027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:47:43.281027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:47:43.281027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:47:43.281027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:47:43.281027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 07 03:47:43.281027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:43.281027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 252/509] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:43 2022] GET /identity/v3/users/cinder => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:43.281027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3aedc36-7471-474a-8b87-e336e33b444e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:43.311974 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:43.341719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:43.353122 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71662aae-eba4-4808-8475-453a29a8ae1d None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:43.383067 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3aedc36-7471-474a-8b87-e336e33b444e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:43.386769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e933273-788d-4659-8669-459cec1b6cb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:43.387405 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e933273-788d-4659-8669-459cec1b6cb7 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:43.387685 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e933273-788d-4659-8669-459cec1b6cb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:43.387962 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e933273-788d-4659-8669-459cec1b6cb7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:43.398333 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e933273-788d-4659-8669-459cec1b6cb7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:43.399793 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e933273-788d-4659-8669-459cec1b6cb7 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:43.414064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-9e933273-788d-4659-8669-459cec1b6cb7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:43.414064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:43.414064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:43.414064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:43.414064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:43.414064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:43.414064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:43.414064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:43.414064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:43.414064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:43.414064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:43.414064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:43.414064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:43.414064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:43.414064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:43.414064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:43.414970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:43.415789 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:43.415789 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:43.415789 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:43.415789 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 258/510] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:43.416009 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e933273-788d-4659-8669-459cec1b6cb7 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:43.420653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40104b40-c6e4-4938-a966-cf2f525d0695 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:43.421382 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40104b40-c6e4-4938-a966-cf2f525d0695 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:43.421667 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40104b40-c6e4-4938-a966-cf2f525d0695 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:43.422076 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40104b40-c6e4-4938-a966-cf2f525d0695 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:43.429481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40104b40-c6e4-4938-a966-cf2f525d0695 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:43.430940 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40104b40-c6e4-4938-a966-cf2f525d0695 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:43.442763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-40104b40-c6e4-4938-a966-cf2f525d0695 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 07 03:47:43.442763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:43.442763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:43.442763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:43.442763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:43.442763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:43.442763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:43.442763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:43.442763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:43.442763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:43.442763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:43.442763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:43.442763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:47:43.442763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:47:43.442763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:47:43.442763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:43.443662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:43.445169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:43.445169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:43.445169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:47:43.445169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:47:43.445169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:47:43.445169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:47:43.445169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:47:43.445169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:47:43.445169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 07 03:47:43.445169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:43.445169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 253/511] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:43 2022] GET /identity/v3/users/glance => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:43.445169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40104b40-c6e4-4938-a966-cf2f525d0695 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:43.522901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e933273-788d-4659-8669-459cec1b6cb7 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:43.536148 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:43.543392 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40104b40-c6e4-4938-a966-cf2f525d0695 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:43.559058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7247d5e8-0dab-4f30-9c49-dcc37791ff3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:43.559670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7247d5e8-0dab-4f30-9c49-dcc37791ff3e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:43.559954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7247d5e8-0dab-4f30-9c49-dcc37791ff3e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:43.560238 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7247d5e8-0dab-4f30-9c49-dcc37791ff3e None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:43.561150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7247d5e8-0dab-4f30-9c49-dcc37791ff3e None admin] RBAC: Adding query filter params (name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:43.561466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7247d5e8-0dab-4f30-9c49-dcc37791ff3e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:43.562888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7247d5e8-0dab-4f30-9c49-dcc37791ff3e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:43.564246 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:43.571481 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 259/512] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:47:43 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:43.576876 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7247d5e8-0dab-4f30-9c49-dcc37791ff3e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:43.579450 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0022328-f5ab-4268-8e3c-af3f8940f36f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:43.580060 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0022328-f5ab-4268-8e3c-af3f8940f36f None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:43.580341 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0022328-f5ab-4268-8e3c-af3f8940f36f None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:43.580617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0022328-f5ab-4268-8e3c-af3f8940f36f None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:43.581262 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0022328-f5ab-4268-8e3c-af3f8940f36f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:43.582767 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0022328-f5ab-4268-8e3c-af3f8940f36f None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:43.604125 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 254/513] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:47:43 2022] POST /identity/v3/users => generated 247 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:47:43.604942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0022328-f5ab-4268-8e3c-af3f8940f36f None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:43.694911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7247d5e8-0dab-4f30-9c49-dcc37791ff3e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:43.731691 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498f27d6-7785-4acb-9ed6-a332f82b0ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:43.732357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498f27d6-7785-4acb-9ed6-a332f82b0ab4 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:43.733275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498f27d6-7785-4acb-9ed6-a332f82b0ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:43.733744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498f27d6-7785-4acb-9ed6-a332f82b0ab4 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:43.734673 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498f27d6-7785-4acb-9ed6-a332f82b0ab4 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:43.734984 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498f27d6-7785-4acb-9ed6-a332f82b0ab4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:43.736559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498f27d6-7785-4acb-9ed6-a332f82b0ab4 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:43.744408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0022328-f5ab-4268-8e3c-af3f8940f36f None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:43.783118 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 260/514] 10.222.0.27 () {56 vars in 1137 bytes} [Wed Sep 7 03:47:43 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:43.783282 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498f27d6-7785-4acb-9ed6-a332f82b0ab4 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:43.783978 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611ae344-d519-40be-8263-ec5c37721be0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:43.784690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611ae344-d519-40be-8263-ec5c37721be0 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:43.784974 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611ae344-d519-40be-8263-ec5c37721be0 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:43.785252 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611ae344-d519-40be-8263-ec5c37721be0 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:43.785961 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611ae344-d519-40be-8263-ec5c37721be0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:43.786309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611ae344-d519-40be-8263-ec5c37721be0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:43.788271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611ae344-d519-40be-8263-ec5c37721be0 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:43.801041 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 255/515] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:43.801578 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611ae344-d519-40be-8263-ec5c37721be0 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:43.817955 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:43.846736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:43.901739 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498f27d6-7785-4acb-9ed6-a332f82b0ab4 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:43.918316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611ae344-d519-40be-8263-ec5c37721be0 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:43.937561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15bacc7-e75e-451e-ac62-4b039380d722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:43.938205 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15bacc7-e75e-451e-ac62-4b039380d722 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:43.938478 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15bacc7-e75e-451e-ac62-4b039380d722 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:43.938755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15bacc7-e75e-451e-ac62-4b039380d722 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:43.939499 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15bacc7-e75e-451e-ac62-4b039380d722 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:43.939809 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15bacc7-e75e-451e-ac62-4b039380d722 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:43.941277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15bacc7-e75e-451e-ac62-4b039380d722 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:43.955699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d88759-ec41-40e7-a0f4-13d2e5b3e93c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:43.956443 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d88759-ec41-40e7-a0f4-13d2e5b3e93c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:43.956719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d88759-ec41-40e7-a0f4-13d2e5b3e93c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:43.956987 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d88759-ec41-40e7-a0f4-13d2e5b3e93c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:43.959871 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 261/516] 10.222.0.27 () {56 vars in 1137 bytes} [Wed Sep 7 03:47:43 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:43.960520 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15bacc7-e75e-451e-ac62-4b039380d722 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:43.971068 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d88759-ec41-40e7-a0f4-13d2e5b3e93c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:43.972968 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d88759-ec41-40e7-a0f4-13d2e5b3e93c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:43.990959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-e4d88759-ec41-40e7-a0f4-13d2e5b3e93c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:43.990959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:43.990959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:43.990959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:43.990959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:43.990959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:43.990959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:43.990959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:43.990959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:43.990959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:43.990959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:43.990959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:43.990959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:43.990959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:43.990959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:43.990959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:43.991878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:43.992713 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:43.992713 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:43.992713 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:43.994487 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 256/517] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:43.995600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d88759-ec41-40e7-a0f4-13d2e5b3e93c None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:44.061444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:44.071610 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15bacc7-e75e-451e-ac62-4b039380d722 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:44.095948 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:44.109403 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62992ba9-a446-40d8-9a77-2ee8f05b55ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:44.109869 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d88759-ec41-40e7-a0f4-13d2e5b3e93c None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:44.110074 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62992ba9-a446-40d8-9a77-2ee8f05b55ac None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:44.110396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62992ba9-a446-40d8-9a77-2ee8f05b55ac None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:44.110672 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62992ba9-a446-40d8-9a77-2ee8f05b55ac None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:44.149769 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2204236c-58d3-4a53-bdde-65bbb4983e66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:44.150528 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2204236c-58d3-4a53-bdde-65bbb4983e66 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:44.150813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2204236c-58d3-4a53-bdde-65bbb4983e66 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:44.151087 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2204236c-58d3-4a53-bdde-65bbb4983e66 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:44.158381 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2204236c-58d3-4a53-bdde-65bbb4983e66 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:44.160142 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2204236c-58d3-4a53-bdde-65bbb4983e66 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:44.172448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-2204236c-58d3-4a53-bdde-65bbb4983e66 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:44.172448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:44.172448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:44.172448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:44.172448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:44.172448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:44.172448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:44.172448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:44.172448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:44.172448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:44.172448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:44.172448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:44.172448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:47:44.172448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:47:44.172448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:44.173328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:44.174823 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:47:44.174823 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:47:44.174823 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:47:44.174823 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:47:44.174823 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:47:44.174823 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:47:44.174823 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:44.174823 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:44.174823 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 257/518] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:44.174823 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2204236c-58d3-4a53-bdde-65bbb4983e66 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:44.186511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62992ba9-a446-40d8-9a77-2ee8f05b55ac None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:44.187980 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62992ba9-a446-40d8-9a77-2ee8f05b55ac None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:44.289365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:44.290350 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2204236c-58d3-4a53-bdde-65bbb4983e66 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:44.305567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 262/519] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:44.305567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62992ba9-a446-40d8-9a77-2ee8f05b55ac None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:44.331839 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566c2755-5436-4e1b-9db8-19924be9d72d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:44.332451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566c2755-5436-4e1b-9db8-19924be9d72d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:44.332734 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566c2755-5436-4e1b-9db8-19924be9d72d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:44.333011 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566c2755-5436-4e1b-9db8-19924be9d72d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:44.352633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:44.409523 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62992ba9-a446-40d8-9a77-2ee8f05b55ac None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:44.410158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566c2755-5436-4e1b-9db8-19924be9d72d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:44.411633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566c2755-5436-4e1b-9db8-19924be9d72d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:44.445654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b37c4b2-cd78-41a1-b757-3e2d8ff36a85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:44.446306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b37c4b2-cd78-41a1-b757-3e2d8ff36a85 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:44.446740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b37c4b2-cd78-41a1-b757-3e2d8ff36a85 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:44.447011 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b37c4b2-cd78-41a1-b757-3e2d8ff36a85 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:44.448423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b37c4b2-cd78-41a1-b757-3e2d8ff36a85 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:44.448733 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b37c4b2-cd78-41a1-b757-3e2d8ff36a85 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:44.450107 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b37c4b2-cd78-41a1-b757-3e2d8ff36a85 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:44.469191 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 263/520] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:44.469191 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b37c4b2-cd78-41a1-b757-3e2d8ff36a85 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:44.514219 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 258/521] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:44.519492 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566c2755-5436-4e1b-9db8-19924be9d72d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:44.528164 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:44.571260 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:44.589771 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b37c4b2-cd78-41a1-b757-3e2d8ff36a85 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:44.623333 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566c2755-5436-4e1b-9db8-19924be9d72d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:44.623951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9a6eea-18fd-4645-8ab0-2423530f7297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:44.624780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9a6eea-18fd-4645-8ab0-2423530f7297 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:44.625264 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9a6eea-18fd-4645-8ab0-2423530f7297 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:44.625755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9a6eea-18fd-4645-8ab0-2423530f7297 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:44.626504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9a6eea-18fd-4645-8ab0-2423530f7297 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:44.626819 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9a6eea-18fd-4645-8ab0-2423530f7297 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:44.628256 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9a6eea-18fd-4645-8ab0-2423530f7297 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:44.651092 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 264/522] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:47:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:44.651092 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9a6eea-18fd-4645-8ab0-2423530f7297 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:44.665379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f201326-a49c-4d30-bc1a-401578f97291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:44.666067 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f201326-a49c-4d30-bc1a-401578f97291 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:44.666387 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f201326-a49c-4d30-bc1a-401578f97291 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:44.666665 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f201326-a49c-4d30-bc1a-401578f97291 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:44.677940 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f201326-a49c-4d30-bc1a-401578f97291 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:44.679435 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f201326-a49c-4d30-bc1a-401578f97291 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:44.694241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-6f201326-a49c-4d30-bc1a-401578f97291 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:44.694241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:44.694241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:44.694241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:44.694241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:44.694241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:44.694241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:44.694241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:44.694241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:44.694241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:44.694241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:44.694241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:44.694241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:44.694241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:44.694241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:44.694241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:44.695321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:44.696445 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:44.696445 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:44.696445 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:44.696445 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 259/523] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:44.696445 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f201326-a49c-4d30-bc1a-401578f97291 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:44.763608 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9a6eea-18fd-4645-8ab0-2423530f7297 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:44.774456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:44.800274 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81d5f13-4b49-47c4-9984-e73526822b9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:44.800917 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81d5f13-4b49-47c4-9984-e73526822b9d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:44.801193 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81d5f13-4b49-47c4-9984-e73526822b9d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:44.801469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81d5f13-4b49-47c4-9984-e73526822b9d None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:44.801780 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:44.805537 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f201326-a49c-4d30-bc1a-401578f97291 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:44.809333 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81d5f13-4b49-47c4-9984-e73526822b9d None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:44.811561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81d5f13-4b49-47c4-9984-e73526822b9d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:44.824214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-e81d5f13-4b49-47c4-9984-e73526822b9d None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 07 03:47:44.824214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:44.824214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:44.824214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:44.824214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:44.824214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:44.824214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:44.824214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:44.824214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:44.824214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:44.824214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:44.824214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:44.824214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:47:44.824214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:47:44.824214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:47:44.824214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:44.825119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:44.826031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:44.826031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:44.826031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:47:44.826031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:47:44.826031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:47:44.826031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:47:44.826031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:47:44.826031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:47:44.826031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 07 03:47:44.826031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:44.826031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 265/524] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:44 2022] GET /identity/v3/users/neutron => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:44.827283 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81d5f13-4b49-47c4-9984-e73526822b9d None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:44.849420 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0098d929-1c9f-4edc-9127-2afb64c0b8d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:44.849976 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0098d929-1c9f-4edc-9127-2afb64c0b8d0 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:44.850296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0098d929-1c9f-4edc-9127-2afb64c0b8d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:44.850578 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0098d929-1c9f-4edc-9127-2afb64c0b8d0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:44.861560 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0098d929-1c9f-4edc-9127-2afb64c0b8d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:44.863052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0098d929-1c9f-4edc-9127-2afb64c0b8d0 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:44.878077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-0098d929-1c9f-4edc-9127-2afb64c0b8d0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:44.878077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:44.878077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:44.878077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:44.878077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:44.878077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:44.878077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:44.878077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:44.878077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:44.878077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:44.878077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:44.878077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:44.878077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:44.878077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:44.878077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:44.878077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:44.879227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:44.880232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:44.880232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:44.880232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:44.880232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 260/525] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:44.880460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0098d929-1c9f-4edc-9127-2afb64c0b8d0 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:44.946204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81d5f13-4b49-47c4-9984-e73526822b9d None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:44.987031 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0098d929-1c9f-4edc-9127-2afb64c0b8d0 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:44.987467 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d434d6e-97b5-4ea6-994e-3661896e5379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:44.988092 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d434d6e-97b5-4ea6-994e-3661896e5379 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:44.988375 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d434d6e-97b5-4ea6-994e-3661896e5379 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:44.988653 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d434d6e-97b5-4ea6-994e-3661896e5379 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:44.989381 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d434d6e-97b5-4ea6-994e-3661896e5379 None admin] RBAC: Adding query filter params (role.id=a7b16bb097f14f78a7a31c04bd6d32bd, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=953cb6c24bcc4551b91ce540d18cd751) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:44.989696 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d434d6e-97b5-4ea6-994e-3661896e5379 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:44.991174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d434d6e-97b5-4ea6-994e-3661896e5379 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:44.996526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 266/526] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:47:44 2022] GET /identity/v3/role_assignments?user.id=953cb6c24bcc4551b91ce540d18cd751&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=a7b16bb097f14f78a7a31c04bd6d32bd => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:44.997312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d434d6e-97b5-4ea6-994e-3661896e5379 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:45.040300 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76d8114-f729-45d8-878c-152ca4a183ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:45.041637 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:45.042049 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76d8114-f729-45d8-878c-152ca4a183ca None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:45.044254 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76d8114-f729-45d8-878c-152ca4a183ca None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:45.044802 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76d8114-f729-45d8-878c-152ca4a183ca None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:45.045754 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76d8114-f729-45d8-878c-152ca4a183ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:45.046316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76d8114-f729-45d8-878c-152ca4a183ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:45.047916 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76d8114-f729-45d8-878c-152ca4a183ca None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:45.067346 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 261/527] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:45.068855 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76d8114-f729-45d8-878c-152ca4a183ca None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:45.091420 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:45.124679 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d434d6e-97b5-4ea6-994e-3661896e5379 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:45.164608 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dcae2d-aab9-4a66-970d-c944ee8a5c1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:45.165502 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dcae2d-aab9-4a66-970d-c944ee8a5c1f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:45.165893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dcae2d-aab9-4a66-970d-c944ee8a5c1f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:45.167068 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dcae2d-aab9-4a66-970d-c944ee8a5c1f None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:45.167378 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dcae2d-aab9-4a66-970d-c944ee8a5c1f None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:45.167795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dcae2d-aab9-4a66-970d-c944ee8a5c1f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:45.169760 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dcae2d-aab9-4a66-970d-c944ee8a5c1f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:45.193200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 267/528] 10.222.0.27 () {56 vars in 1139 bytes} [Wed Sep 7 03:47:44 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:45.194729 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dcae2d-aab9-4a66-970d-c944ee8a5c1f None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:45.212617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76d8114-f729-45d8-878c-152ca4a183ca None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:45.249503 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a29e8d8-1e6c-4d1f-8521-8bd434ef2497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:45.250137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a29e8d8-1e6c-4d1f-8521-8bd434ef2497 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:45.250440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a29e8d8-1e6c-4d1f-8521-8bd434ef2497 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:45.250721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a29e8d8-1e6c-4d1f-8521-8bd434ef2497 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:45.251369 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a29e8d8-1e6c-4d1f-8521-8bd434ef2497 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:45.251676 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a29e8d8-1e6c-4d1f-8521-8bd434ef2497 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:45.253047 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a29e8d8-1e6c-4d1f-8521-8bd434ef2497 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:45.265865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 262/529] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:45.266852 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a29e8d8-1e6c-4d1f-8521-8bd434ef2497 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:45.312050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:45.323882 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dcae2d-aab9-4a66-970d-c944ee8a5c1f None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:45.340145 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:45.367322 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed16605-17c1-4d15-a551-5c406ae98c9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:45.368251 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed16605-17c1-4d15-a551-5c406ae98c9c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:45.368697 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed16605-17c1-4d15-a551-5c406ae98c9c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:45.369206 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed16605-17c1-4d15-a551-5c406ae98c9c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:45.372834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a29e8d8-1e6c-4d1f-8521-8bd434ef2497 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:45.377357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed16605-17c1-4d15-a551-5c406ae98c9c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:45.379210 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed16605-17c1-4d15-a551-5c406ae98c9c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:45.393811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-0ed16605-17c1-4d15-a551-5c406ae98c9c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:45.393811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:45.393811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:45.393811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:45.393811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:45.393811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:45.393811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:45.393811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:45.393811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:45.393811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:45.393811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:45.393811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:45.393811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:47:45.393811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:47:45.393811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:45.394901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:45.395908 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:47:45.395908 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:47:45.395908 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:47:45.395908 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:47:45.395908 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:47:45.395908 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:47:45.395908 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:45.395908 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:45.402394 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 268/530] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:45.403594 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed16605-17c1-4d15-a551-5c406ae98c9c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:45.410313 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3c9239-8292-4a3b-b48c-a819d82cdfa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:45.410937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3c9239-8292-4a3b-b48c-a819d82cdfa0 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:45.411221 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3c9239-8292-4a3b-b48c-a819d82cdfa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:45.411499 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3c9239-8292-4a3b-b48c-a819d82cdfa0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:45.490510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3c9239-8292-4a3b-b48c-a819d82cdfa0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:45.492127 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3c9239-8292-4a3b-b48c-a819d82cdfa0 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:45.533477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed16605-17c1-4d15-a551-5c406ae98c9c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:45.577038 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:45.578239 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a9e19a-71f0-4e23-a1c8-a1fceb5ffce6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:45.579289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a9e19a-71f0-4e23-a1c8-a1fceb5ffce6 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:45.579573 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a9e19a-71f0-4e23-a1c8-a1fceb5ffce6 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:45.579843 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a9e19a-71f0-4e23-a1c8-a1fceb5ffce6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:45.586937 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a9e19a-71f0-4e23-a1c8-a1fceb5ffce6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:45.588391 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a9e19a-71f0-4e23-a1c8-a1fceb5ffce6 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:45.600477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-73a9e19a-71f0-4e23-a1c8-a1fceb5ffce6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:45.600477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:45.600477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:45.600477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:45.600477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:45.600477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:45.600477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:45.600477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:45.600477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:45.600477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:45.600477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:45.600477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:45.600477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:47:45.600477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:47:45.600477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:45.601320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:45.602191 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:47:45.602191 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:47:45.602191 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:47:45.602191 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:47:45.602191 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:47:45.602191 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:47:45.602191 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:45.602191 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:45.602191 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 269/531] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:45.603110 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a9e19a-71f0-4e23-a1c8-a1fceb5ffce6 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:45.618025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 263/532] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:45.618529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:45.620244 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3c9239-8292-4a3b-b48c-a819d82cdfa0 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:45.736423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3c9239-8292-4a3b-b48c-a819d82cdfa0 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:45.738292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a9e19a-71f0-4e23-a1c8-a1fceb5ffce6 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:45.776997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3843722-0afe-467c-b0a3-3c3d2690c242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:45.777741 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3843722-0afe-467c-b0a3-3c3d2690c242 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:45.777921 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3843722-0afe-467c-b0a3-3c3d2690c242 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:45.782104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3843722-0afe-467c-b0a3-3c3d2690c242 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:45.782104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3843722-0afe-467c-b0a3-3c3d2690c242 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:45.782104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3843722-0afe-467c-b0a3-3c3d2690c242 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:45.782104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3843722-0afe-467c-b0a3-3c3d2690c242 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:45.786562 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb02b89-42d5-454e-a66d-bc9a43a97db9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:45.787349 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb02b89-42d5-454e-a66d-bc9a43a97db9 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:45.787778 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb02b89-42d5-454e-a66d-bc9a43a97db9 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:45.788199 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb02b89-42d5-454e-a66d-bc9a43a97db9 None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:45.789032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb02b89-42d5-454e-a66d-bc9a43a97db9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:45.789506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb02b89-42d5-454e-a66d-bc9a43a97db9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:45.791717 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb02b89-42d5-454e-a66d-bc9a43a97db9 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:45.796359 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 270/533] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:47:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:45.797467 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3843722-0afe-467c-b0a3-3c3d2690c242 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:45.817085 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 264/534] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:47:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:45.818679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb02b89-42d5-454e-a66d-bc9a43a97db9 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:45.861955 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:45.877502 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:45.911111 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3843722-0afe-467c-b0a3-3c3d2690c242 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:45.937456 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb02b89-42d5-454e-a66d-bc9a43a97db9 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:45.948585 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb627354-e3c7-4b6b-82fb-7a1cba6f0cea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:45.949224 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb627354-e3c7-4b6b-82fb-7a1cba6f0cea None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:45.949505 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb627354-e3c7-4b6b-82fb-7a1cba6f0cea None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:45.949774 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb627354-e3c7-4b6b-82fb-7a1cba6f0cea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:45.961288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb627354-e3c7-4b6b-82fb-7a1cba6f0cea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:45.962790 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb627354-e3c7-4b6b-82fb-7a1cba6f0cea None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:45.972786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0be9006-afce-49db-aebc-6b383643e7b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:45.973390 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0be9006-afce-49db-aebc-6b383643e7b4 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:45.973667 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0be9006-afce-49db-aebc-6b383643e7b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:45.973945 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0be9006-afce-49db-aebc-6b383643e7b4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:45.974731 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0be9006-afce-49db-aebc-6b383643e7b4 None admin] RBAC: Adding query filter params (role.id=a7b16bb097f14f78a7a31c04bd6d32bd, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=989081d67c8945dc8d853bb2769c902a) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:45.975041 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0be9006-afce-49db-aebc-6b383643e7b4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:45.976460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0be9006-afce-49db-aebc-6b383643e7b4 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:45.978554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-bb627354-e3c7-4b6b-82fb-7a1cba6f0cea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:45.978554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:45.978554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:45.978554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:45.978554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:45.978554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:45.978554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:45.978554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:45.978554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:45.978554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:45.978554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:45.978554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:45.978554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:45.978554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:45.978554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:45.978554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:45.979416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:45.980252 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:45.980252 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:45.980252 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:45.980252 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 271/535] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:45.980887 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb627354-e3c7-4b6b-82fb-7a1cba6f0cea None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:45.990302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 265/536] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:47:45 2022] GET /identity/v3/role_assignments?user.id=989081d67c8945dc8d853bb2769c902a&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=a7b16bb097f14f78a7a31c04bd6d32bd => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:45.990302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05440e6b-e63d-4344-86ce-b5c7250b1795 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:45.990302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05440e6b-e63d-4344-86ce-b5c7250b1795 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:45.990689 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05440e6b-e63d-4344-86ce-b5c7250b1795 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:46.001919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 266/537] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:46.001919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05440e6b-e63d-4344-86ce-b5c7250b1795 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:46.110761 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb627354-e3c7-4b6b-82fb-7a1cba6f0cea None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:46.138362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:46.142353 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:46.152472 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05440e6b-e63d-4344-86ce-b5c7250b1795 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:46.154790 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1a99cd-8241-474c-b68e-be53096dc348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:46.155813 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1a99cd-8241-474c-b68e-be53096dc348 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:46.156197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1a99cd-8241-474c-b68e-be53096dc348 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:46.156578 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1a99cd-8241-474c-b68e-be53096dc348 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:46.157513 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1a99cd-8241-474c-b68e-be53096dc348 None admin] RBAC: Adding query filter params (role.id=a7b16bb097f14f78a7a31c04bd6d32bd, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=dfa09be69ee3444e94f00cc139bf50e7) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:46.157926 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1a99cd-8241-474c-b68e-be53096dc348 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:46.159550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1a99cd-8241-474c-b68e-be53096dc348 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:46.170855 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 272/538] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:47:45 2022] GET /identity/v3/role_assignments?user.id=dfa09be69ee3444e94f00cc139bf50e7&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=a7b16bb097f14f78a7a31c04bd6d32bd => generated 259 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:46.172651 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3817d721-c270-4789-a248-8f26c7e0fe5e None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:46.173038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3817d721-c270-4789-a248-8f26c7e0fe5e None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:46.173413 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3817d721-c270-4789-a248-8f26c7e0fe5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:46.196406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f151318b-5c4c-4fe3-b703-3055b7819b29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:46.197288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f151318b-5c4c-4fe3-b703-3055b7819b29 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:46.197708 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f151318b-5c4c-4fe3-b703-3055b7819b29 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:46.198183 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f151318b-5c4c-4fe3-b703-3055b7819b29 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:46.199008 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f151318b-5c4c-4fe3-b703-3055b7819b29 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:46.199458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f151318b-5c4c-4fe3-b703-3055b7819b29 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:46.203148 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f151318b-5c4c-4fe3-b703-3055b7819b29 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:46.209210 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-3817d721-c270-4789-a248-8f26c7e0fe5e None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:46.210082 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-3817d721-c270-4789-a248-8f26c7e0fe5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:46.221614 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 267/539] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:46.227117 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f151318b-5c4c-4fe3-b703-3055b7819b29 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:46.319562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 273/540] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:46.359365 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f151318b-5c4c-4fe3-b703-3055b7819b29 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:46.397805 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a09c8b-ec0e-4bd5-bc24-560618a17a53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:46.398471 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a09c8b-ec0e-4bd5-bc24-560618a17a53 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:46.398746 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a09c8b-ec0e-4bd5-bc24-560618a17a53 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:46.399020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a09c8b-ec0e-4bd5-bc24-560618a17a53 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:46.407721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a09c8b-ec0e-4bd5-bc24-560618a17a53 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:46.409195 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a09c8b-ec0e-4bd5-bc24-560618a17a53 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:46.421752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-c3a09c8b-ec0e-4bd5-bc24-560618a17a53 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:46.421752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:46.421752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:46.421752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:46.421752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:46.421752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:46.421752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:46.421752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:46.421752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:46.421752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:46.421752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:46.421752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:46.421752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:47:46.421752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:47:46.421752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:46.422661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:46.423504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:47:46.423504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:47:46.423504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:47:46.423504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:47:46.423504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:47:46.423504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:47:46.423504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:46.423504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:46.423504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 268/541] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:46.428886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3817d721-c270-4789-a248-8f26c7e0fe5e None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:46.580767 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3817d721-c270-4789-a248-8f26c7e0fe5e None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:46.608583 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:46.630608 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9248e47-bbea-4b84-918c-718790f4e1c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:46.631221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9248e47-bbea-4b84-918c-718790f4e1c1 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:46.631500 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9248e47-bbea-4b84-918c-718790f4e1c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:46.631772 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9248e47-bbea-4b84-918c-718790f4e1c1 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:46.632467 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9248e47-bbea-4b84-918c-718790f4e1c1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:46.632773 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9248e47-bbea-4b84-918c-718790f4e1c1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:46.634232 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9248e47-bbea-4b84-918c-718790f4e1c1 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:46.651011 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 274/542] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:47:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:46.657495 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a09c8b-ec0e-4bd5-bc24-560618a17a53 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:46.687854 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:46.766029 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9248e47-bbea-4b84-918c-718790f4e1c1 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:46.792468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a09c8b-ec0e-4bd5-bc24-560618a17a53 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:46.834229 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7891335e-31d7-49da-9fce-ce72c5186e7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:46.834749 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7891335e-31d7-49da-9fce-ce72c5186e7c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:46.835025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7891335e-31d7-49da-9fce-ce72c5186e7c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:46.835296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7891335e-31d7-49da-9fce-ce72c5186e7c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:46.836021 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7891335e-31d7-49da-9fce-ce72c5186e7c None admin] RBAC: Adding query filter params (role.id=a7b16bb097f14f78a7a31c04bd6d32bd, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=77aac66a6ab44fc1a80ff2fd1572254b) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:46.836326 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7891335e-31d7-49da-9fce-ce72c5186e7c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:46.837769 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7891335e-31d7-49da-9fce-ce72c5186e7c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:46.845430 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 269/543] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:47:46 2022] GET /identity/v3/role_assignments?user.id=77aac66a6ab44fc1a80ff2fd1572254b&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=a7b16bb097f14f78a7a31c04bd6d32bd => generated 259 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:46.958002 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9248e47-bbea-4b84-918c-718790f4e1c1 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:47.005023 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5946a8-b02e-4900-ba61-1406dc5744cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:47.005753 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5946a8-b02e-4900-ba61-1406dc5744cb None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:47.006037 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5946a8-b02e-4900-ba61-1406dc5744cb None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:47.006328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5946a8-b02e-4900-ba61-1406dc5744cb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:47.018818 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5946a8-b02e-4900-ba61-1406dc5744cb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:47.024397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-8b5946a8-b02e-4900-ba61-1406dc5744cb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:47.024397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:47.024397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:47.024397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:47.024397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:47.024397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:47.024397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:47.024397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:47.024397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:47.024397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:47.024397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:47.024397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:47.024397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:47.024397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:47.024397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:47.024397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:47.026614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:47.026614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:47.026614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:47.026614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:47.026614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:47.026614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:47.026614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:47.026614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:47.026614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:47.026614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:47.026614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:47.026614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:47.026614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:47.026614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:47.026614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:47.026614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:47.028561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:47.028561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:47.028561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:47.028561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:47.028561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:47.028561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:47.028561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:47.028561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:47.028561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:47.028561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:47.028561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:47.028561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:47.028561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 275/544] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:47.036375 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7891335e-31d7-49da-9fce-ce72c5186e7c None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:47.158919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:47.179288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7891335e-31d7-49da-9fce-ce72c5186e7c None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:47.217803 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccba751a-56fa-4199-8e80-95c3ceff72d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:47.218807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccba751a-56fa-4199-8e80-95c3ceff72d4 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:47.219341 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccba751a-56fa-4199-8e80-95c3ceff72d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:47.219842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccba751a-56fa-4199-8e80-95c3ceff72d4 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:47.220727 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccba751a-56fa-4199-8e80-95c3ceff72d4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:47.221241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccba751a-56fa-4199-8e80-95c3ceff72d4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:47.222872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccba751a-56fa-4199-8e80-95c3ceff72d4 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:47.239795 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 270/545] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:47:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:47.245938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5946a8-b02e-4900-ba61-1406dc5744cb None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:47.318645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:47.399217 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d441a1d4-5d10-4701-aea2-bb13303ee71c None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:47.399849 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d441a1d4-5d10-4701-aea2-bb13303ee71c None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:47.400346 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d441a1d4-5d10-4701-aea2-bb13303ee71c None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:47.401908 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 271/546] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:47:47.408648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442a6de4-609c-48e6-8b4b-14c4926d6a2f None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:47.409247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442a6de4-609c-48e6-8b4b-14c4926d6a2f None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:47.409726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442a6de4-609c-48e6-8b4b-14c4926d6a2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:47.433333 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-442a6de4-609c-48e6-8b4b-14c4926d6a2f None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:47.434838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-442a6de4-609c-48e6-8b4b-14c4926d6a2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:47.437720 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5946a8-b02e-4900-ba61-1406dc5744cb None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:47.507829 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be960092-f34a-4752-a11d-0a6fd66a63d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:47.508480 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be960092-f34a-4752-a11d-0a6fd66a63d0 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:47.508761 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be960092-f34a-4752-a11d-0a6fd66a63d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:47.509036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be960092-f34a-4752-a11d-0a6fd66a63d0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:47.531077 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be960092-f34a-4752-a11d-0a6fd66a63d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:47.532571 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be960092-f34a-4752-a11d-0a6fd66a63d0 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:47.552282 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 272/547] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:47.558686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-be960092-f34a-4752-a11d-0a6fd66a63d0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:47.558686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:47.558686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:47.558686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:47.558686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:47.558686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:47.558686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:47.558686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:47.558686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:47.558686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:47.558686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:47.558686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:47.558686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:47.558686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:47.558686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:47.558686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:47.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:47.560406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:47.560406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:47.560406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:47.560406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 276/548] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:47.565675 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442a6de4-609c-48e6-8b4b-14c4926d6a2f None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:47.656293 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:47.691994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442a6de4-609c-48e6-8b4b-14c4926d6a2f None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:47.726789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e8a18f-a124-4acd-b5c3-ba233028096d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:47.727389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e8a18f-a124-4acd-b5c3-ba233028096d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:47.727664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e8a18f-a124-4acd-b5c3-ba233028096d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:47.727930 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e8a18f-a124-4acd-b5c3-ba233028096d None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:47.728560 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e8a18f-a124-4acd-b5c3-ba233028096d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:47.728855 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e8a18f-a124-4acd-b5c3-ba233028096d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:47.730237 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e8a18f-a124-4acd-b5c3-ba233028096d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:47.742728 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 273/549] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:47.747861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be960092-f34a-4752-a11d-0a6fd66a63d0 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:47.865124 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:47.882789 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be960092-f34a-4752-a11d-0a6fd66a63d0 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:47.930969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1c5da4-1aa2-4874-b899-7e4a6a143b3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:47.931623 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1c5da4-1aa2-4874-b899-7e4a6a143b3b None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:47.931914 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1c5da4-1aa2-4874-b899-7e4a6a143b3b None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:47.932199 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1c5da4-1aa2-4874-b899-7e4a6a143b3b None admin] PATH_INFO: `/v3/users/swift` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:47.940775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1c5da4-1aa2-4874-b899-7e4a6a143b3b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:47.942313 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1c5da4-1aa2-4874-b899-7e4a6a143b3b None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:47.954545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-bc1c5da4-1aa2-4874-b899-7e4a6a143b3b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 07 03:47:47.954545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:47.954545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:47.954545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:47.954545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:47.954545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:47.954545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:47.954545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:47.954545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:47.954545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:47.954545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:47.954545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:47.954545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:47:47.954545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:47:47.954545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:47:47.954545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:47.956042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:47.957032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:47.957032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:47.957032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:47:47.957032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:47:47.957032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:47:47.957032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:47:47.957032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:47:47.957032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:47:47.957032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 07 03:47:47.957032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:47.957032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 277/550] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:47:47 2022] GET /identity/v3/users/swift => generated 83 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:47.961645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e8a18f-a124-4acd-b5c3-ba233028096d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:48.052196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1c5da4-1aa2-4874-b899-7e4a6a143b3b None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:48.102751 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e8a18f-a124-4acd-b5c3-ba233028096d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:48.156697 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:48.163437 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4c4cde-af17-40cc-92f4-6dd3b4dfca3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:48.164340 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4c4cde-af17-40cc-92f4-6dd3b4dfca3c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:48.164632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4c4cde-af17-40cc-92f4-6dd3b4dfca3c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:48.164915 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4c4cde-af17-40cc-92f4-6dd3b4dfca3c None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:48.166395 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4c4cde-af17-40cc-92f4-6dd3b4dfca3c None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:48.166723 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4c4cde-af17-40cc-92f4-6dd3b4dfca3c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:48.168354 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4c4cde-af17-40cc-92f4-6dd3b4dfca3c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:48.188925 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 274/551] 10.222.0.27 () {56 vars in 1135 bytes} [Wed Sep 7 03:47:47 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:48.194819 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4c4cde-af17-40cc-92f4-6dd3b4dfca3c None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:48.259474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1c5da4-1aa2-4874-b899-7e4a6a143b3b None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:48.329554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6fb879-313e-4f66-9919-4c0b01570679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:48.330281 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6fb879-313e-4f66-9919-4c0b01570679 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:48.330568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6fb879-313e-4f66-9919-4c0b01570679 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:48.330842 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6fb879-313e-4f66-9919-4c0b01570679 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:48.354307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6fb879-313e-4f66-9919-4c0b01570679 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:48.365109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-1e6fb879-313e-4f66-9919-4c0b01570679 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:48.365109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:48.365109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:48.365109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:48.365109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:48.365109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:48.365109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:48.365109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:48.365109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:48.365109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:48.365109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:48.365109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:48.365109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:48.365109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:48.365109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:48.365109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:48.366228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:48.366228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:48.366228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:48.366228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:48.366228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:48.366228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:48.366228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:48.366228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:48.366228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:48.366228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:48.366228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:48.366228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:48.366228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:48.366228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:48.366228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:48.366228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:48.367025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:48.367025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:48.367025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:48.367025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:48.367025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:48.367025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:48.367025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:48.367025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:48.367025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:48.367025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:48.367025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:48.367025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:48.369490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 278/552] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:48.376115 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6fb879-313e-4f66-9919-4c0b01570679 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:48.402776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:48.405199 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4c4cde-af17-40cc-92f4-6dd3b4dfca3c None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:48.491037 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e033d3f-b8bf-4b4f-b74f-26fc5ce62064 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:48.491574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e033d3f-b8bf-4b4f-b74f-26fc5ce62064 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:48.491856 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e033d3f-b8bf-4b4f-b74f-26fc5ce62064 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:48.492127 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e033d3f-b8bf-4b4f-b74f-26fc5ce62064 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:48.531052 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6fb879-313e-4f66-9919-4c0b01570679 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:48.570435 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e9d521-caf7-478b-b2c8-d5d89bf4f920 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:48.571058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e9d521-caf7-478b-b2c8-d5d89bf4f920 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:48.571384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e9d521-caf7-478b-b2c8-d5d89bf4f920 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:48.571667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e9d521-caf7-478b-b2c8-d5d89bf4f920 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:48.572720 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e9d521-caf7-478b-b2c8-d5d89bf4f920 None admin] RBAC: Adding query filter params (name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:48.573033 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e9d521-caf7-478b-b2c8-d5d89bf4f920 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:48.574779 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e9d521-caf7-478b-b2c8-d5d89bf4f920 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:48.585730 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 279/553] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:47:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:48.591310 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e9d521-caf7-478b-b2c8-d5d89bf4f920 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:48.607032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:48.619669 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e033d3f-b8bf-4b4f-b74f-26fc5ce62064 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:48.621422 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e033d3f-b8bf-4b4f-b74f-26fc5ce62064 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:48.684841 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:48.727390 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e9d521-caf7-478b-b2c8-d5d89bf4f920 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:48.728569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 275/554] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 534 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 07 03:47:48.730382 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ae01cb-1864-42c1-b973-9c0d258cab87 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:48.730869 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ae01cb-1864-42c1-b973-9c0d258cab87 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:48.731397 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ae01cb-1864-42c1-b973-9c0d258cab87 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:48.737036 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 276/555] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:48.738079 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ae01cb-1864-42c1-b973-9c0d258cab87 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:48.768945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5d61aa-e1bd-4df4-96b4-0697cc4d6dc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:48.769599 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5d61aa-e1bd-4df4-96b4-0697cc4d6dc2 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:48.769777 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5d61aa-e1bd-4df4-96b4-0697cc4d6dc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:48.770051 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5d61aa-e1bd-4df4-96b4-0697cc4d6dc2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:48.781773 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5d61aa-e1bd-4df4-96b4-0697cc4d6dc2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:48.783294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5d61aa-e1bd-4df4-96b4-0697cc4d6dc2 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:48.798891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-fe5d61aa-e1bd-4df4-96b4-0697cc4d6dc2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:48.798891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:48.798891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:48.798891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:48.798891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:48.798891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:48.798891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:48.798891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:48.798891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:48.798891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:48.798891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:48.798891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:48.798891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:48.798891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:48.798891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:48.798891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:48.799875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:48.800729 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:48.800729 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:48.800729 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:48.804812 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 280/556] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:48.806064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d97a26b-5bcb-4a27-a6a1-7bda9b218193 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:48.806642 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d97a26b-5bcb-4a27-a6a1-7bda9b218193 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:48.806926 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d97a26b-5bcb-4a27-a6a1-7bda9b218193 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:48.826296 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-9d97a26b-5bcb-4a27-a6a1-7bda9b218193 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:48.826703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-9d97a26b-5bcb-4a27-a6a1-7bda9b218193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:48.851764 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:48.865305 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ae01cb-1864-42c1-b973-9c0d258cab87 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:48.906246 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 281/557] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:48.906246 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d97a26b-5bcb-4a27-a6a1-7bda9b218193 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:48.909828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97837e42-6f3a-4c34-aff8-a479e20fada5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:48.910847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97837e42-6f3a-4c34-aff8-a479e20fada5 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:48.911327 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97837e42-6f3a-4c34-aff8-a479e20fada5 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:48.911815 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97837e42-6f3a-4c34-aff8-a479e20fada5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:48.930081 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97837e42-6f3a-4c34-aff8-a479e20fada5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:48.932008 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97837e42-6f3a-4c34-aff8-a479e20fada5 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:48.962505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-97837e42-6f3a-4c34-aff8-a479e20fada5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:48.962505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:48.962505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:48.962505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:48.962505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:48.962505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:48.962505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:48.962505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:48.962505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:48.962505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:48.962505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:48.962505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:48.962505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:48.962505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:48.962505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:48.962505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:48.963936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:48.964841 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:48.964841 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:48.964841 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:48.965007 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:48.973015 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 277/558] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:48.979359 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97837e42-6f3a-4c34-aff8-a479e20fada5 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:49.019498 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d97a26b-5bcb-4a27-a6a1-7bda9b218193 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:49.057768 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74568d9-948c-4063-8656-8730136b076d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:49.058648 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74568d9-948c-4063-8656-8730136b076d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:49.058929 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74568d9-948c-4063-8656-8730136b076d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:49.059584 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74568d9-948c-4063-8656-8730136b076d None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:49.060239 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74568d9-948c-4063-8656-8730136b076d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:49.060600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74568d9-948c-4063-8656-8730136b076d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:49.062980 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74568d9-948c-4063-8656-8730136b076d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:49.079938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 282/559] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:49.081244 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6a5d50-5b56-44a0-9044-86aea9f86ffb None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:49.081536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6a5d50-5b56-44a0-9044-86aea9f86ffb None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:49.081812 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6a5d50-5b56-44a0-9044-86aea9f86ffb None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:49.086199 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 283/560] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:49.087009 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6a5d50-5b56-44a0-9044-86aea9f86ffb None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:49.127284 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97837e42-6f3a-4c34-aff8-a479e20fada5 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:49.172632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df58cf3f-053e-42c9-ad68-22861d20dbc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:49.173502 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df58cf3f-053e-42c9-ad68-22861d20dbc7 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:49.173776 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df58cf3f-053e-42c9-ad68-22861d20dbc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:49.174326 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df58cf3f-053e-42c9-ad68-22861d20dbc7 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:49.175377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df58cf3f-053e-42c9-ad68-22861d20dbc7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:49.175683 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df58cf3f-053e-42c9-ad68-22861d20dbc7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:49.177575 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df58cf3f-053e-42c9-ad68-22861d20dbc7 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:49.194248 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 278/561] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:49.195553 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190c0aa9-4d1e-474d-8c35-08118914963a None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:49.195839 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190c0aa9-4d1e-474d-8c35-08118914963a None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:49.196112 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190c0aa9-4d1e-474d-8c35-08118914963a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:49.205035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:49.223003 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-190c0aa9-4d1e-474d-8c35-08118914963a None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:49.224834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-190c0aa9-4d1e-474d-8c35-08118914963a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:49.231848 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6a5d50-5b56-44a0-9044-86aea9f86ffb None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:49.249423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:49.282567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421bfb94-8212-4bee-8620-2ddd6ea2a59d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:49.283735 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421bfb94-8212-4bee-8620-2ddd6ea2a59d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:49.283735 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421bfb94-8212-4bee-8620-2ddd6ea2a59d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:49.283735 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421bfb94-8212-4bee-8620-2ddd6ea2a59d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:49.291614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421bfb94-8212-4bee-8620-2ddd6ea2a59d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:49.293064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421bfb94-8212-4bee-8620-2ddd6ea2a59d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:49.304448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-421bfb94-8212-4bee-8620-2ddd6ea2a59d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:47:49.304448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:49.304448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:49.304448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:49.304448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:49.304448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:49.304448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:49.304448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:49.304448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:49.304448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:49.304448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:49.304448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:49.304448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:47:49.304448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:47:49.304448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:47:49.304448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:49.305420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:49.307101 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:49.307101 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:49.307101 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:47:49.307101 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:47:49.307101 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:47:49.307101 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:47:49.307101 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:47:49.307101 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:47:49.307101 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:47:49.307101 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:49.307101 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 284/562] 10.222.0.27 () {56 vars in 1093 bytes} [Wed Sep 7 03:47:49 2022] GET /identity/v3/users/nova => generated 82 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:49.307826 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421bfb94-8212-4bee-8620-2ddd6ea2a59d None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:49.323449 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 279/563] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:49.330344 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190c0aa9-4d1e-474d-8c35-08118914963a None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:49.429669 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190c0aa9-4d1e-474d-8c35-08118914963a None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:49.438447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421bfb94-8212-4bee-8620-2ddd6ea2a59d None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:49.460360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:49.466935 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e456f5-6f23-4d19-973a-4e55e87914ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:49.467809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e456f5-6f23-4d19-973a-4e55e87914ed None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:49.468177 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e456f5-6f23-4d19-973a-4e55e87914ed None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:49.468445 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e456f5-6f23-4d19-973a-4e55e87914ed None admin] PATH_INFO: `/v3/roles/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:49.475774 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fc2816-b1e6-4bfb-8d30-c3f443359f73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:49.476602 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fc2816-b1e6-4bfb-8d30-c3f443359f73 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:49.477076 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fc2816-b1e6-4bfb-8d30-c3f443359f73 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:49.477525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fc2816-b1e6-4bfb-8d30-c3f443359f73 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:49.479028 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e456f5-6f23-4d19-973a-4e55e87914ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:49.484167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-f2e456f5-6f23-4d19-973a-4e55e87914ed None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:49.484167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:49.484167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:49.484167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:49.484167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:49.484167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:49.484167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:49.484167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:49.484167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:49.484167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:49.484167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:49.484167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:49.484167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:49.484167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:49.484167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:49.484167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:49.485082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:49.485082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:49.485082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:49.485082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:49.485082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:49.485082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:49.485082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:49.485082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:49.485082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:49.485082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:49.485082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:49.485082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:49.485082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:49.485082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:49.485082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:49.485082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:49.485919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:49.485919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:49.485919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:49.485919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:49.485919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:49.485919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:49.485919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:49.485919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:49.485919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:49.485919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:49.485919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:49.485919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:49.485919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 280/564] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:49.486654 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e456f5-6f23-4d19-973a-4e55e87914ed None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:49.487612 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fc2816-b1e6-4bfb-8d30-c3f443359f73 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:49.489218 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fc2816-b1e6-4bfb-8d30-c3f443359f73 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:49.501836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-61fc2816-b1e6-4bfb-8d30-c3f443359f73 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:49.501836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:49.501836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:49.501836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:49.501836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:49.501836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:49.501836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:49.501836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:49.501836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:49.501836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:49.501836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:49.501836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:49.501836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:47:49.501836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:47:49.501836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:49.502890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:49.503769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:47:49.503769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:47:49.503769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:47:49.503769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:47:49.503769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:47:49.503769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:47:49.503769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:49.503769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:49.503769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 285/565] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:49.504244 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1338affa-9f9e-490d-bfb6-88eac405f409 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:49.504322 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1338affa-9f9e-490d-bfb6-88eac405f409 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:49.504895 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1338affa-9f9e-490d-bfb6-88eac405f409 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:49.505937 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 286/566] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:49 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:49.510458 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1338affa-9f9e-490d-bfb6-88eac405f409 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:49.548054 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:49.605314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e456f5-6f23-4d19-973a-4e55e87914ed None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:49.619477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1338affa-9f9e-490d-bfb6-88eac405f409 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:49.641705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c48408-4641-4130-b183-75b494d1b793 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:49.642343 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c48408-4641-4130-b183-75b494d1b793 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:49.642625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c48408-4641-4130-b183-75b494d1b793 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:49.642899 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c48408-4641-4130-b183-75b494d1b793 None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:49.643623 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c48408-4641-4130-b183-75b494d1b793 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:49.643930 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c48408-4641-4130-b183-75b494d1b793 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:49.645363 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c48408-4641-4130-b183-75b494d1b793 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:49.653849 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67449d86-9816-4ab3-a696-a73c1f030e53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:49.654507 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67449d86-9816-4ab3-a696-a73c1f030e53 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:49.654791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67449d86-9816-4ab3-a696-a73c1f030e53 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:49.655068 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67449d86-9816-4ab3-a696-a73c1f030e53 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:49.655728 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67449d86-9816-4ab3-a696-a73c1f030e53 None admin] RBAC: Adding query filter params (name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:49.656036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67449d86-9816-4ab3-a696-a73c1f030e53 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:49.657401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67449d86-9816-4ab3-a696-a73c1f030e53 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:49.662721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 281/567] 10.222.0.27 () {56 vars in 1133 bytes} [Wed Sep 7 03:47:49 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:49.663622 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c48408-4641-4130-b183-75b494d1b793 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:49.669630 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 287/568] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:47:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:49.669630 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d193c485-1694-44e6-8111-8830796d82c5 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:49.669630 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d193c485-1694-44e6-8111-8830796d82c5 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:49.669630 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d193c485-1694-44e6-8111-8830796d82c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:49.684765 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-d193c485-1694-44e6-8111-8830796d82c5 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:49.686506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-d193c485-1694-44e6-8111-8830796d82c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:49.719837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:49.755162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 288/569] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:49.755996 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d193c485-1694-44e6-8111-8830796d82c5 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:49.769039 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c48408-4641-4130-b183-75b494d1b793 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:49.800536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:49.805702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa65fb5-b99e-4ae3-8c59-f73e0fe3ac68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:49.806568 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa65fb5-b99e-4ae3-8c59-f73e0fe3ac68 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:49.806853 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa65fb5-b99e-4ae3-8c59-f73e0fe3ac68 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:49.807128 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa65fb5-b99e-4ae3-8c59-f73e0fe3ac68 None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:49.807940 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa65fb5-b99e-4ae3-8c59-f73e0fe3ac68 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:49.808144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa65fb5-b99e-4ae3-8c59-f73e0fe3ac68 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:49.809563 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa65fb5-b99e-4ae3-8c59-f73e0fe3ac68 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:49.822497 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 282/570] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:47:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:49.823326 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa65fb5-b99e-4ae3-8c59-f73e0fe3ac68 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:49.891450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d193c485-1694-44e6-8111-8830796d82c5 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:49.927941 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa65fb5-b99e-4ae3-8c59-f73e0fe3ac68 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:49.930419 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce42d5a-458f-4c0e-a953-89d5d2456900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:49.931131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce42d5a-458f-4c0e-a953-89d5d2456900 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:49.931412 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce42d5a-458f-4c0e-a953-89d5d2456900 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:49.931684 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce42d5a-458f-4c0e-a953-89d5d2456900 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:49.939039 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:49.943996 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce42d5a-458f-4c0e-a953-89d5d2456900 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:49.948944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-cce42d5a-458f-4c0e-a953-89d5d2456900 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:49.948944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:49.948944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:49.948944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:49.948944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:49.948944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:49.948944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:49.948944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:49.948944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:49.948944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:49.948944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:49.948944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:49.948944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:49.948944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:49.948944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:49.948944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:49.950050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:49.950050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:49.950050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:49.950050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:49.950050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:49.950050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:49.950050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:49.950050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:49.950050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:49.950050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:49.950050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:49.950050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:49.950050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:49.950050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:49.950050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:49.950050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:49.950933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:49.950933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:49.950933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:49.950933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:49.950933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:49.950933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:49.950933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:49.950933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:49.950933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:49.950933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:49.950933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:49.950933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:49.950933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 289/571] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:49.956498 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce42d5a-458f-4c0e-a953-89d5d2456900 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:49.967171 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942e4685-5193-4c7d-be8f-dbf1a23ba606 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:49.967803 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942e4685-5193-4c7d-be8f-dbf1a23ba606 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:49.968085 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942e4685-5193-4c7d-be8f-dbf1a23ba606 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:49.968358 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942e4685-5193-4c7d-be8f-dbf1a23ba606 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:49.979177 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942e4685-5193-4c7d-be8f-dbf1a23ba606 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:49.980626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942e4685-5193-4c7d-be8f-dbf1a23ba606 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:49.995152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-942e4685-5193-4c7d-be8f-dbf1a23ba606 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:49.995152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:49.995152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:49.995152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:49.995152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:49.995152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:49.995152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:49.995152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:49.995152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:49.995152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:49.995152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:49.995152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:49.995152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:49.995152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:49.995152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:49.995152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:49.996095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:49.997085 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:49.997085 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:49.997085 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:49.997085 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 283/572] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:49.997085 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942e4685-5193-4c7d-be8f-dbf1a23ba606 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:50.049883 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:50.065854 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce42d5a-458f-4c0e-a953-89d5d2456900 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:50.100598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b25bfe4-2628-4b2c-845c-104d9d4e4345 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:50.100775 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942e4685-5193-4c7d-be8f-dbf1a23ba606 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:50.101225 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b25bfe4-2628-4b2c-845c-104d9d4e4345 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:50.101506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b25bfe4-2628-4b2c-845c-104d9d4e4345 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:50.101782 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b25bfe4-2628-4b2c-845c-104d9d4e4345 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:50.111988 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b25bfe4-2628-4b2c-845c-104d9d4e4345 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:50.113440 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b25bfe4-2628-4b2c-845c-104d9d4e4345 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:50.128751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-5b25bfe4-2628-4b2c-845c-104d9d4e4345 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:50.128751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:50.128751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:50.128751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:50.128751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:50.128751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:50.128751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:50.128751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:50.128751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:50.128751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:50.128751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:50.128751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:50.128751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:50.128751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:50.128751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:50.128751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:50.129894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:50.130975 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:50.130975 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:50.130975 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:50.130975 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 290/573] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:50.130975 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b25bfe4-2628-4b2c-845c-104d9d4e4345 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:50.136190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f28bd1-1867-4aa8-9507-ce6f357484cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:50.136786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f28bd1-1867-4aa8-9507-ce6f357484cf None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:50.137062 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f28bd1-1867-4aa8-9507-ce6f357484cf None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:50.137330 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f28bd1-1867-4aa8-9507-ce6f357484cf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:50.138049 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f28bd1-1867-4aa8-9507-ce6f357484cf None admin] RBAC: Adding query filter params (role.id=a7b16bb097f14f78a7a31c04bd6d32bd, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=f5c3ab9728f74cd8bb2614bcca9e8619) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:50.138389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f28bd1-1867-4aa8-9507-ce6f357484cf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:50.140141 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f28bd1-1867-4aa8-9507-ce6f357484cf None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:50.145863 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 284/574] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:47:49 2022] GET /identity/v3/role_assignments?user.id=f5c3ab9728f74cd8bb2614bcca9e8619&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=a7b16bb097f14f78a7a31c04bd6d32bd => generated 259 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:50.146755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f28bd1-1867-4aa8-9507-ce6f357484cf None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:50.178632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:50.253749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b25bfe4-2628-4b2c-845c-104d9d4e4345 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:50.273074 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f28bd1-1867-4aa8-9507-ce6f357484cf None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:50.294015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df846e84-1877-48a2-a95e-dc37f05d81f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:50.294847 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df846e84-1877-48a2-a95e-dc37f05d81f3 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:50.295202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df846e84-1877-48a2-a95e-dc37f05d81f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:50.295553 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df846e84-1877-48a2-a95e-dc37f05d81f3 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:50.296303 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df846e84-1877-48a2-a95e-dc37f05d81f3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:50.296680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df846e84-1877-48a2-a95e-dc37f05d81f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:50.298176 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df846e84-1877-48a2-a95e-dc37f05d81f3 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:50.303933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:50.314014 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 291/575] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:47:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:50.314014 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df846e84-1877-48a2-a95e-dc37f05d81f3 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:50.321675 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2573766-2bc9-416a-be23-f6cc63a51f06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:50.322273 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2573766-2bc9-416a-be23-f6cc63a51f06 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:50.322639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2573766-2bc9-416a-be23-f6cc63a51f06 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:50.322921 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2573766-2bc9-416a-be23-f6cc63a51f06 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:50.324018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2573766-2bc9-416a-be23-f6cc63a51f06 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:50.324018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2573766-2bc9-416a-be23-f6cc63a51f06 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:50.325250 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2573766-2bc9-416a-be23-f6cc63a51f06 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:50.337810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 285/576] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:50.338745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2573766-2bc9-416a-be23-f6cc63a51f06 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:50.424752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:50.426560 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df846e84-1877-48a2-a95e-dc37f05d81f3 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:50.443300 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2573766-2bc9-416a-be23-f6cc63a51f06 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:50.465593 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fbacc0-95f6-4a06-b90e-f92fa1c3511e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:50.466271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fbacc0-95f6-4a06-b90e-f92fa1c3511e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:50.466551 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fbacc0-95f6-4a06-b90e-f92fa1c3511e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:50.467507 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fbacc0-95f6-4a06-b90e-f92fa1c3511e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:50.479084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fbacc0-95f6-4a06-b90e-f92fa1c3511e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:50.479355 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7b928f-808e-4bfc-823a-d2ef07870401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:50.479955 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7b928f-808e-4bfc-823a-d2ef07870401 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:50.480230 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7b928f-808e-4bfc-823a-d2ef07870401 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:50.480498 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7b928f-808e-4bfc-823a-d2ef07870401 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:50.481133 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7b928f-808e-4bfc-823a-d2ef07870401 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:50.481431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7b928f-808e-4bfc-823a-d2ef07870401 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:50.483785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7b928f-808e-4bfc-823a-d2ef07870401 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:50.484702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-68fbacc0-95f6-4a06-b90e-f92fa1c3511e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:50.484702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:50.484702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:50.484702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:50.484702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:50.484702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:50.484702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:50.484702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:50.484702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:50.484702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:50.484702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:50.484702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:50.484702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:50.484702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:50.484702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:50.484702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:50.485568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:50.485568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:50.485568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:50.485568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:50.485568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:50.485568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:50.485568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:50.485568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:50.485568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:50.485568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:50.485568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:50.485568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:50.485568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:50.485568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:50.485568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:50.485568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:50.486423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:50.486423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:50.486423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:50.486423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:50.486423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:50.486423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:50.486423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:50.486423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:50.486423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:50.486423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:50.486423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:50.486423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:50.486423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 292/577] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:50.487099 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fbacc0-95f6-4a06-b90e-f92fa1c3511e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:50.503584 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 286/578] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:50.507341 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7b928f-808e-4bfc-823a-d2ef07870401 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:50.554292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:50.604772 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fbacc0-95f6-4a06-b90e-f92fa1c3511e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:50.618955 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7b928f-808e-4bfc-823a-d2ef07870401 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:50.643324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b805fa-e7cc-4b11-bcb2-52f0cafc759a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:50.644060 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b805fa-e7cc-4b11-bcb2-52f0cafc759a None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:50.644370 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b805fa-e7cc-4b11-bcb2-52f0cafc759a None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:50.644699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b805fa-e7cc-4b11-bcb2-52f0cafc759a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:50.655255 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2ed082-7653-443b-a97e-ee1c7c70914d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:50.655851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2ed082-7653-443b-a97e-ee1c7c70914d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:50.655959 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b805fa-e7cc-4b11-bcb2-52f0cafc759a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:50.656136 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2ed082-7653-443b-a97e-ee1c7c70914d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:50.656395 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2ed082-7653-443b-a97e-ee1c7c70914d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:50.657356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b805fa-e7cc-4b11-bcb2-52f0cafc759a None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:50.663331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2ed082-7653-443b-a97e-ee1c7c70914d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:50.664748 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2ed082-7653-443b-a97e-ee1c7c70914d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:50.672611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-98b805fa-e7cc-4b11-bcb2-52f0cafc759a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:50.672611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:50.672611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:50.672611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:50.672611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:50.672611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:50.672611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:50.672611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:50.672611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:50.672611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:50.672611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:50.672611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:50.672611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:50.672611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:50.672611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:50.672611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:50.673506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:50.674805 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:50.674805 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:50.674805 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:50.676523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-fe2ed082-7653-443b-a97e-ee1c7c70914d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:50.676523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:50.676523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:50.676523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:50.676523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:50.676523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:50.676523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:50.676523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:50.676523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:50.676523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:50.676523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:50.676523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:50.676523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:47:50.676523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:47:50.676523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:50.677401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:50.678320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:47:50.678320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:47:50.678320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:47:50.678320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:47:50.678320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:47:50.678320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:47:50.678320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:50.678320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:50.678320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 287/579] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:50.678881 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2ed082-7653-443b-a97e-ee1c7c70914d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:50.682565 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 293/580] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:50.683367 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b805fa-e7cc-4b11-bcb2-52f0cafc759a None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:50.689243 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:50.786854 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2ed082-7653-443b-a97e-ee1c7c70914d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:50.787482 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b805fa-e7cc-4b11-bcb2-52f0cafc759a None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:50.801920 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:50.823838 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5860ef5-1d78-4bee-b9a1-6ab496055a1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:50.824593 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5860ef5-1d78-4bee-b9a1-6ab496055a1e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:50.824846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5860ef5-1d78-4bee-b9a1-6ab496055a1e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:50.825120 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5860ef5-1d78-4bee-b9a1-6ab496055a1e None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:50.827324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02e9296-7be2-4feb-b24c-717b84f980f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:50.827883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02e9296-7be2-4feb-b24c-717b84f980f6 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:50.828163 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02e9296-7be2-4feb-b24c-717b84f980f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:50.828432 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02e9296-7be2-4feb-b24c-717b84f980f6 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:50.829359 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02e9296-7be2-4feb-b24c-717b84f980f6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:50.829648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02e9296-7be2-4feb-b24c-717b84f980f6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:50.831036 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02e9296-7be2-4feb-b24c-717b84f980f6 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:50.834217 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5860ef5-1d78-4bee-b9a1-6ab496055a1e None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:50.835668 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5860ef5-1d78-4bee-b9a1-6ab496055a1e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:50.845176 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 288/581] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:47:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:50.845176 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02e9296-7be2-4feb-b24c-717b84f980f6 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:50.847639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-a5860ef5-1d78-4bee-b9a1-6ab496055a1e None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 07 03:47:50.847639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:50.847639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:50.847639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:50.847639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:50.847639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:50.847639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:50.847639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:50.847639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:50.847639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:50.847639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:50.847639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:50.847639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:47:50.847639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:47:50.847639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:47:50.847639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:50.848555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:50.849491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:50.849491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:50.849491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:47:50.849491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:47:50.849491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:47:50.849491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:47:50.849491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:47:50.849491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:47:50.849491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 07 03:47:50.849491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:50.849491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 294/582] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:50 2022] GET /identity/v3/users/cinder => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:50.850075 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5860ef5-1d78-4bee-b9a1-6ab496055a1e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:50.929174 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:50.959676 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5860ef5-1d78-4bee-b9a1-6ab496055a1e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:50.967873 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02e9296-7be2-4feb-b24c-717b84f980f6 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:50.996265 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cf0a72-67a1-4083-ad48-65db3d2cf9a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:50.996879 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cf0a72-67a1-4083-ad48-65db3d2cf9a8 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:50.997174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cf0a72-67a1-4083-ad48-65db3d2cf9a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:50.997495 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cf0a72-67a1-4083-ad48-65db3d2cf9a8 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:50.998877 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cf0a72-67a1-4083-ad48-65db3d2cf9a8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:50.999285 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cf0a72-67a1-4083-ad48-65db3d2cf9a8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:51.001066 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cf0a72-67a1-4083-ad48-65db3d2cf9a8 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:51.001566 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab808d0-251e-44f2-80ce-61199bf6ed8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:51.002204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab808d0-251e-44f2-80ce-61199bf6ed8c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:51.002496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab808d0-251e-44f2-80ce-61199bf6ed8c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:51.002766 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab808d0-251e-44f2-80ce-61199bf6ed8c None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:51.003459 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab808d0-251e-44f2-80ce-61199bf6ed8c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:51.003777 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab808d0-251e-44f2-80ce-61199bf6ed8c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:51.005178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab808d0-251e-44f2-80ce-61199bf6ed8c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:51.014808 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 295/583] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:51.015929 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cf0a72-67a1-4083-ad48-65db3d2cf9a8 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:51.022082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 289/584] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:47:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:51.023356 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab808d0-251e-44f2-80ce-61199bf6ed8c None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:51.052078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:51.127079 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cf0a72-67a1-4083-ad48-65db3d2cf9a8 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:51.129967 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab808d0-251e-44f2-80ce-61199bf6ed8c None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:51.157805 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:51.163583 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1804d9a4-8968-4ea9-8aba-6e5008727e1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:51.164229 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1804d9a4-8968-4ea9-8aba-6e5008727e1a None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:51.164512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1804d9a4-8968-4ea9-8aba-6e5008727e1a None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:51.164789 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1804d9a4-8968-4ea9-8aba-6e5008727e1a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:51.166981 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4ad714-4d81-45cb-bb1d-c0b2002d03ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:51.167889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4ad714-4d81-45cb-bb1d-c0b2002d03ca None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:51.168174 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4ad714-4d81-45cb-bb1d-c0b2002d03ca None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:51.168445 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4ad714-4d81-45cb-bb1d-c0b2002d03ca None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:51.169176 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4ad714-4d81-45cb-bb1d-c0b2002d03ca None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:51.169480 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4ad714-4d81-45cb-bb1d-c0b2002d03ca None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:51.171010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4ad714-4d81-45cb-bb1d-c0b2002d03ca None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:51.175361 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1804d9a4-8968-4ea9-8aba-6e5008727e1a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:51.176806 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1804d9a4-8968-4ea9-8aba-6e5008727e1a None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:51.195233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 290/585] 10.222.0.27 () {56 vars in 1137 bytes} [Wed Sep 7 03:47:51 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:51.195233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4ad714-4d81-45cb-bb1d-c0b2002d03ca None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:51.195445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-1804d9a4-8968-4ea9-8aba-6e5008727e1a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:51.195445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:51.195445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:51.195445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:51.195445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:51.195445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:51.195445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:51.195445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:51.195445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:51.195445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:51.195445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:51.195445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:51.195445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:51.195445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:51.195445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:51.195445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:51.196302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:51.197174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:51.197174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:51.197174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:51.198159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 296/586] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:51.198801 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1804d9a4-8968-4ea9-8aba-6e5008727e1a None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:51.296621 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:51.304911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4ad714-4d81-45cb-bb1d-c0b2002d03ca None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:51.314917 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1804d9a4-8968-4ea9-8aba-6e5008727e1a None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:51.341187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7160296-7f37-43fd-bd9d-75eb9ebae671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:51.342009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7160296-7f37-43fd-bd9d-75eb9ebae671 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:51.342350 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7160296-7f37-43fd-bd9d-75eb9ebae671 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:51.342731 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7160296-7f37-43fd-bd9d-75eb9ebae671 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:51.350182 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79d5a2c-9580-4ff7-8e20-a5aa85609c60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:51.350812 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79d5a2c-9580-4ff7-8e20-a5aa85609c60 None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:51.351096 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79d5a2c-9580-4ff7-8e20-a5aa85609c60 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:51.351375 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79d5a2c-9580-4ff7-8e20-a5aa85609c60 None admin] PATH_INFO: `/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/953cb6c24bcc4551b91ce540d18cd751/roles/a7b16bb097f14f78a7a31c04bd6d32bd` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:51.351836 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7160296-7f37-43fd-bd9d-75eb9ebae671 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:51.353260 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7160296-7f37-43fd-bd9d-75eb9ebae671 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:51.367421 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79d5a2c-9580-4ff7-8e20-a5aa85609c60 None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4816e95144485e8135e5b8f68c3662, user_id=953cb6c24bcc4551b91ce540d18cd751, role_id=a7b16bb097f14f78a7a31c04bd6d32bd)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:51.368318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-f7160296-7f37-43fd-bd9d-75eb9ebae671 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 07 03:47:51.368318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:51.368318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:51.368318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:51.368318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:51.368318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:51.368318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:51.368318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:51.368318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:51.368318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:51.368318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:51.368318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:51.368318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:47:51.368318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:47:51.368318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:47:51.368318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:51.369208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:51.370158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:51.370158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:51.370158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:47:51.370158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:47:51.370158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:47:51.370158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:47:51.370158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:47:51.370158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:47:51.370158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 07 03:47:51.370158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:51.371195 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79d5a2c-9580-4ff7-8e20-a5aa85609c60 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:51.374048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 291/587] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:51 2022] GET /identity/v3/users/glance => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:51.375167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7160296-7f37-43fd-bd9d-75eb9ebae671 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:51.385017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 297/588] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:47:51 2022] PUT /identity/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/953cb6c24bcc4551b91ce540d18cd751/roles/a7b16bb097f14f78a7a31c04bd6d32bd => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:47:51.385833 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79d5a2c-9580-4ff7-8e20-a5aa85609c60 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:51.452009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:51.511730 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79d5a2c-9580-4ff7-8e20-a5aa85609c60 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:51.533393 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7160296-7f37-43fd-bd9d-75eb9ebae671 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:51.553442 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e60a0fd-3154-48b5-9967-f5726dfec725 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:51.554074 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e60a0fd-3154-48b5-9967-f5726dfec725 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:51.554396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e60a0fd-3154-48b5-9967-f5726dfec725 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:51.554673 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e60a0fd-3154-48b5-9967-f5726dfec725 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:51.555343 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e60a0fd-3154-48b5-9967-f5726dfec725 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:51.555655 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e60a0fd-3154-48b5-9967-f5726dfec725 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:51.557618 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e60a0fd-3154-48b5-9967-f5726dfec725 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:51.561037 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:51.575243 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 298/589] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:51.576759 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e60a0fd-3154-48b5-9967-f5726dfec725 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:51.586487 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966fb6e5-b78a-4e9c-a6f0-012b7328b3ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:51.587146 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966fb6e5-b78a-4e9c-a6f0-012b7328b3ec None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:51.587402 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966fb6e5-b78a-4e9c-a6f0-012b7328b3ec None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:51.587676 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966fb6e5-b78a-4e9c-a6f0-012b7328b3ec None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:51.600492 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966fb6e5-b78a-4e9c-a6f0-012b7328b3ec None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:51.602313 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966fb6e5-b78a-4e9c-a6f0-012b7328b3ec None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:51.619616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-966fb6e5-b78a-4e9c-a6f0-012b7328b3ec None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:51.619616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:51.619616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:51.619616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:51.619616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:51.619616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:51.619616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:51.619616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:51.619616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:51.619616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:51.619616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:51.619616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:51.619616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:51.619616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:51.619616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:51.619616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:51.620807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:51.621721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:51.621721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:51.621721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:51.624098 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 292/590] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:51.625038 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966fb6e5-b78a-4e9c-a6f0-012b7328b3ec None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:51.716165 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e60a0fd-3154-48b5-9967-f5726dfec725 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:51.747415 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:51.753749 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966fb6e5-b78a-4e9c-a6f0-012b7328b3ec None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:51.754777 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964718e3-5f4b-441a-a0c5-234acd6074c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:51.755621 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964718e3-5f4b-441a-a0c5-234acd6074c0 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:51.756046 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964718e3-5f4b-441a-a0c5-234acd6074c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:51.756492 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964718e3-5f4b-441a-a0c5-234acd6074c0 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:51.757388 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964718e3-5f4b-441a-a0c5-234acd6074c0 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:51.757852 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964718e3-5f4b-441a-a0c5-234acd6074c0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:51.759451 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964718e3-5f4b-441a-a0c5-234acd6074c0 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:51.785005 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 299/591] 10.222.0.27 () {56 vars in 1137 bytes} [Wed Sep 7 03:47:51 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:51.786229 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964718e3-5f4b-441a-a0c5-234acd6074c0 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:51.795620 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be49c6a5-6877-4a1f-ada0-10a710702429 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:51.796344 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be49c6a5-6877-4a1f-ada0-10a710702429 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:51.796525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be49c6a5-6877-4a1f-ada0-10a710702429 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:51.797375 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be49c6a5-6877-4a1f-ada0-10a710702429 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:51.807934 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be49c6a5-6877-4a1f-ada0-10a710702429 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:51.809793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be49c6a5-6877-4a1f-ada0-10a710702429 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:51.824822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-be49c6a5-6877-4a1f-ada0-10a710702429 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 07 03:47:51.824822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:51.824822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:51.824822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:51.824822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:51.824822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:51.824822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:51.824822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:51.824822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:51.824822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:51.824822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:51.824822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:51.824822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:47:51.824822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:47:51.824822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:47:51.824822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:51.825851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:51.827144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:51.827144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:51.827144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:47:51.827144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:47:51.827144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:47:51.827144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:47:51.827144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:47:51.827144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:47:51.827144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 07 03:47:51.827144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:51.829448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 293/592] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:51 2022] GET /identity/v3/users/neutron => generated 85 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:51.830450 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be49c6a5-6877-4a1f-ada0-10a710702429 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:51.861554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:51.920845 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964718e3-5f4b-441a-a0c5-234acd6074c0 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:51.943785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be49c6a5-6877-4a1f-ada0-10a710702429 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:51.964906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef50ba73-905b-4f90-af96-7e4f017038bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:51.965545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef50ba73-905b-4f90-af96-7e4f017038bf None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:51.965826 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef50ba73-905b-4f90-af96-7e4f017038bf None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:51.966099 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef50ba73-905b-4f90-af96-7e4f017038bf None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:51.967341 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef50ba73-905b-4f90-af96-7e4f017038bf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:51.967593 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef50ba73-905b-4f90-af96-7e4f017038bf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:51.969139 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef50ba73-905b-4f90-af96-7e4f017038bf None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:51.986258 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 300/593] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:51.987165 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef50ba73-905b-4f90-af96-7e4f017038bf None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:51.990381 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c7ff39-6f6f-4836-8437-0a1d3de61046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:51.991494 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c7ff39-6f6f-4836-8437-0a1d3de61046 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:51.991982 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c7ff39-6f6f-4836-8437-0a1d3de61046 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:51.992478 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c7ff39-6f6f-4836-8437-0a1d3de61046 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:52.004991 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c7ff39-6f6f-4836-8437-0a1d3de61046 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:52.007020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c7ff39-6f6f-4836-8437-0a1d3de61046 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:52.023885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-64c7ff39-6f6f-4836-8437-0a1d3de61046 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:52.023885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:52.023885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:52.023885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:52.023885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:52.023885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:52.023885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:52.023885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:52.023885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:52.023885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:52.023885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:52.023885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:52.023885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:52.023885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:52.023885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:52.023885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:52.024866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:52.025747 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:52.025747 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:52.025747 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:52.027974 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 294/594] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:52.029082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c7ff39-6f6f-4836-8437-0a1d3de61046 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:52.043508 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:52.095760 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef50ba73-905b-4f90-af96-7e4f017038bf None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:52.109867 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:52.133105 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d132f5b-3c5a-41e5-96d9-73e1e3ddb70f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:52.133737 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d132f5b-3c5a-41e5-96d9-73e1e3ddb70f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:52.133986 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d132f5b-3c5a-41e5-96d9-73e1e3ddb70f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:52.134360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d132f5b-3c5a-41e5-96d9-73e1e3ddb70f None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:52.135094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d132f5b-3c5a-41e5-96d9-73e1e3ddb70f None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:52.135399 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d132f5b-3c5a-41e5-96d9-73e1e3ddb70f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:52.136839 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d132f5b-3c5a-41e5-96d9-73e1e3ddb70f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:52.151075 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c7ff39-6f6f-4836-8437-0a1d3de61046 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:52.160741 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 301/595] 10.222.0.27 () {56 vars in 1139 bytes} [Wed Sep 7 03:47:51 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:52.160741 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d132f5b-3c5a-41e5-96d9-73e1e3ddb70f None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:52.193604 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a863955b-f25a-486e-ac91-913263e8cd0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:52.194706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a863955b-f25a-486e-ac91-913263e8cd0c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:52.195198 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a863955b-f25a-486e-ac91-913263e8cd0c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:52.195726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a863955b-f25a-486e-ac91-913263e8cd0c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:52.204399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a863955b-f25a-486e-ac91-913263e8cd0c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:52.206334 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a863955b-f25a-486e-ac91-913263e8cd0c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:52.219705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-a863955b-f25a-486e-ac91-913263e8cd0c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:52.219705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:52.219705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:52.219705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:52.219705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:52.219705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:52.219705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:52.219705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:52.219705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:52.219705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:52.219705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:52.219705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:52.219705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:47:52.219705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:47:52.219705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:52.220626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:52.221528 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:47:52.221528 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:47:52.221528 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:47:52.221528 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:47:52.221528 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:47:52.221528 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:47:52.221528 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:52.221528 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:52.224560 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 295/596] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:52.225865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a863955b-f25a-486e-ac91-913263e8cd0c None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:52.272184 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d132f5b-3c5a-41e5-96d9-73e1e3ddb70f None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:52.307603 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6177d6-3e30-4c34-a697-654f79b7e2fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:52.308204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6177d6-3e30-4c34-a697-654f79b7e2fd None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:52.308484 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6177d6-3e30-4c34-a697-654f79b7e2fd None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:52.308756 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6177d6-3e30-4c34-a697-654f79b7e2fd None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:52.309407 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6177d6-3e30-4c34-a697-654f79b7e2fd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:52.309710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6177d6-3e30-4c34-a697-654f79b7e2fd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:52.310573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:52.311573 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6177d6-3e30-4c34-a697-654f79b7e2fd None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:52.325746 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 302/597] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:52.330960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6177d6-3e30-4c34-a697-654f79b7e2fd None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:52.351692 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a863955b-f25a-486e-ac91-913263e8cd0c None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:52.364520 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:52.390054 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918ecb2a-dd44-48d8-8e73-12189826e921 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:52.390962 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918ecb2a-dd44-48d8-8e73-12189826e921 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:52.391434 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918ecb2a-dd44-48d8-8e73-12189826e921 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:52.391778 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918ecb2a-dd44-48d8-8e73-12189826e921 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:52.403825 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918ecb2a-dd44-48d8-8e73-12189826e921 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:52.406787 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918ecb2a-dd44-48d8-8e73-12189826e921 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:52.424704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-918ecb2a-dd44-48d8-8e73-12189826e921 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:52.424704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:52.424704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:52.424704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:52.424704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:52.424704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:52.424704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:52.424704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:52.424704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:52.424704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:52.424704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:52.424704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:52.424704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:52.424704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:52.424704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:52.424704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:52.425661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:52.431743 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:52.431743 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:52.431743 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:52.431743 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 296/598] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:52.431743 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918ecb2a-dd44-48d8-8e73-12189826e921 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:52.434438 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6177d6-3e30-4c34-a697-654f79b7e2fd None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:52.469542 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020b858c-2eca-4d61-9a79-e55f53a0a2f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:52.470231 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020b858c-2eca-4d61-9a79-e55f53a0a2f1 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:52.470459 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020b858c-2eca-4d61-9a79-e55f53a0a2f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:52.470739 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020b858c-2eca-4d61-9a79-e55f53a0a2f1 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:52.471416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020b858c-2eca-4d61-9a79-e55f53a0a2f1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:52.471720 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020b858c-2eca-4d61-9a79-e55f53a0a2f1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:52.473138 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020b858c-2eca-4d61-9a79-e55f53a0a2f1 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:52.485435 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 303/599] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:47:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:52.486631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506476c3-ec3c-4481-84bf-3a2128cc5212 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:52.486915 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506476c3-ec3c-4481-84bf-3a2128cc5212 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:52.487190 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506476c3-ec3c-4481-84bf-3a2128cc5212 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:52.490155 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 304/600] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:52 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:52.490155 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506476c3-ec3c-4481-84bf-3a2128cc5212 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:52.554092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918ecb2a-dd44-48d8-8e73-12189826e921 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:52.576712 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:52.589460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4491b8-707d-4059-bd72-502d0fb39240 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:52.590208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4491b8-707d-4059-bd72-502d0fb39240 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:52.590547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4491b8-707d-4059-bd72-502d0fb39240 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:52.590878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4491b8-707d-4059-bd72-502d0fb39240 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:52.591508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:52.598937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4491b8-707d-4059-bd72-502d0fb39240 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:52.600474 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4491b8-707d-4059-bd72-502d0fb39240 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:52.604538 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506476c3-ec3c-4481-84bf-3a2128cc5212 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:52.612523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-5b4491b8-707d-4059-bd72-502d0fb39240 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:52.612523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:52.612523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:52.612523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:52.612523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:52.612523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:52.612523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:52.612523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:52.612523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:52.612523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:52.612523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:52.612523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:52.612523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:47:52.612523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:47:52.612523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:52.613600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:52.614554 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:47:52.614554 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:47:52.614554 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:47:52.614554 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:47:52.614554 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:47:52.614554 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:47:52.614554 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:52.614554 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:52.614554 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 297/601] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:52.615055 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4491b8-707d-4059-bd72-502d0fb39240 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:52.646195 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf94789f-eea1-4371-b141-30fbaf8693c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:52.646816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf94789f-eea1-4371-b141-30fbaf8693c1 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:52.647100 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf94789f-eea1-4371-b141-30fbaf8693c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:52.647371 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf94789f-eea1-4371-b141-30fbaf8693c1 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:52.648024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf94789f-eea1-4371-b141-30fbaf8693c1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:52.648434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf94789f-eea1-4371-b141-30fbaf8693c1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:52.649695 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf94789f-eea1-4371-b141-30fbaf8693c1 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:52.667896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 305/602] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:52.669796 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9e7de6-e7f3-4032-bd58-75dc1d5cd028 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:52.670309 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9e7de6-e7f3-4032-bd58-75dc1d5cd028 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:52.670783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9e7de6-e7f3-4032-bd58-75dc1d5cd028 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:52.694302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-3c9e7de6-e7f3-4032-bd58-75dc1d5cd028 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:52.695530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-3c9e7de6-e7f3-4032-bd58-75dc1d5cd028 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:52.723217 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4491b8-707d-4059-bd72-502d0fb39240 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:52.761307 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ff4d2c-fbdd-40ec-9334-ff3518101123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:52.762139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ff4d2c-fbdd-40ec-9334-ff3518101123 None admin] REQUEST_METHOD: `PUT` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:52.762557 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ff4d2c-fbdd-40ec-9334-ff3518101123 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:52.762925 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ff4d2c-fbdd-40ec-9334-ff3518101123 None admin] PATH_INFO: `/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/989081d67c8945dc8d853bb2769c902a/roles/a7b16bb097f14f78a7a31c04bd6d32bd` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:52.770377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 306/603] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:52.770377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9e7de6-e7f3-4032-bd58-75dc1d5cd028 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:52.785478 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ff4d2c-fbdd-40ec-9334-ff3518101123 None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4816e95144485e8135e5b8f68c3662, user_id=989081d67c8945dc8d853bb2769c902a, role_id=a7b16bb097f14f78a7a31c04bd6d32bd)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:52.787075 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ff4d2c-fbdd-40ec-9334-ff3518101123 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:52.802950 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 298/604] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:47:52 2022] PUT /identity/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/989081d67c8945dc8d853bb2769c902a/roles/a7b16bb097f14f78a7a31c04bd6d32bd => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:47:52.803988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ff4d2c-fbdd-40ec-9334-ff3518101123 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:52.869766 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:52.941032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:52.964725 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ff4d2c-fbdd-40ec-9334-ff3518101123 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:52.995850 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9e7de6-e7f3-4032-bd58-75dc1d5cd028 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:53.006714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ac8b36-bc48-4fb9-8b1e-7b8987175617 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:53.007521 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ac8b36-bc48-4fb9-8b1e-7b8987175617 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:53.007896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ac8b36-bc48-4fb9-8b1e-7b8987175617 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:53.008272 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ac8b36-bc48-4fb9-8b1e-7b8987175617 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:53.016501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ac8b36-bc48-4fb9-8b1e-7b8987175617 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:53.019701 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ac8b36-bc48-4fb9-8b1e-7b8987175617 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:53.035742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-84ac8b36-bc48-4fb9-8b1e-7b8987175617 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:53.035742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:53.035742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:53.035742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:53.035742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:53.035742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:53.035742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:53.035742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:53.035742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:53.035742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:53.035742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:53.035742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:53.035742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:47:53.035742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:47:53.035742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:53.036722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:53.037831 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:47:53.037831 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:47:53.037831 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:47:53.037831 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:47:53.037831 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:47:53.037831 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:47:53.037831 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:53.037831 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:53.039860 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 299/605] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:53.047458 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab493117-4747-42ae-940b-34547cc1dde7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:53.048253 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab493117-4747-42ae-940b-34547cc1dde7 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:53.048637 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab493117-4747-42ae-940b-34547cc1dde7 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:53.049009 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab493117-4747-42ae-940b-34547cc1dde7 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:53.049864 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab493117-4747-42ae-940b-34547cc1dde7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:53.050312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab493117-4747-42ae-940b-34547cc1dde7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:53.052006 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab493117-4747-42ae-940b-34547cc1dde7 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:53.053439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ac8b36-bc48-4fb9-8b1e-7b8987175617 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:53.065563 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 307/606] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:47:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:53.074012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab493117-4747-42ae-940b-34547cc1dde7 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:53.163235 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:53.174944 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ac8b36-bc48-4fb9-8b1e-7b8987175617 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:53.210799 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab493117-4747-42ae-940b-34547cc1dde7 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:53.212623 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a77028-e111-40d6-9f2f-03dfd7eca01b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:53.213533 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a77028-e111-40d6-9f2f-03dfd7eca01b None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:53.213962 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a77028-e111-40d6-9f2f-03dfd7eca01b None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:53.214412 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a77028-e111-40d6-9f2f-03dfd7eca01b None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:53.215254 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a77028-e111-40d6-9f2f-03dfd7eca01b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:53.215708 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a77028-e111-40d6-9f2f-03dfd7eca01b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:53.218094 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a77028-e111-40d6-9f2f-03dfd7eca01b None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:53.232292 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 300/607] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:47:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:47:53.233612 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a77028-e111-40d6-9f2f-03dfd7eca01b None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:53.246795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b889f371-dc4d-4be4-bd1e-c3a2077deba1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:53.247381 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b889f371-dc4d-4be4-bd1e-c3a2077deba1 None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:53.247656 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b889f371-dc4d-4be4-bd1e-c3a2077deba1 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:53.248498 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b889f371-dc4d-4be4-bd1e-c3a2077deba1 None admin] PATH_INFO: `/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/dfa09be69ee3444e94f00cc139bf50e7/roles/a7b16bb097f14f78a7a31c04bd6d32bd` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:53.260960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:53.267573 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b889f371-dc4d-4be4-bd1e-c3a2077deba1 None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4816e95144485e8135e5b8f68c3662, user_id=dfa09be69ee3444e94f00cc139bf50e7, role_id=a7b16bb097f14f78a7a31c04bd6d32bd)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:53.269121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b889f371-dc4d-4be4-bd1e-c3a2077deba1 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:53.293075 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 308/608] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:47:53 2022] PUT /identity/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/dfa09be69ee3444e94f00cc139bf50e7/roles/a7b16bb097f14f78a7a31c04bd6d32bd => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 07 03:47:53.293075 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b889f371-dc4d-4be4-bd1e-c3a2077deba1 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:53.414347 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a77028-e111-40d6-9f2f-03dfd7eca01b None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:53.432386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b889f371-dc4d-4be4-bd1e-c3a2077deba1 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:53.459162 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036ec85b-9edc-455e-90c6-005731286201 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:53.460063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036ec85b-9edc-455e-90c6-005731286201 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:53.460495 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036ec85b-9edc-455e-90c6-005731286201 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:53.460918 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036ec85b-9edc-455e-90c6-005731286201 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:53.474462 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036ec85b-9edc-455e-90c6-005731286201 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:53.480927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-036ec85b-9edc-455e-90c6-005731286201 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:53.480927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:53.480927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:53.480927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:53.480927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:53.480927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:53.480927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:53.480927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:53.480927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:53.480927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:53.480927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:53.480927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:53.480927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:53.480927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:53.480927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:53.480927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:53.482091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:53.482091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:53.482091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:53.482091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:53.482091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:53.482091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:53.482091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:53.482091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:53.482091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:53.482091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:53.482091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:53.482091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:53.482091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:53.482091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:53.482091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:53.482091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:53.483339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:53.483339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:53.483339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:53.483339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:53.483339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:53.483339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:53.483339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:53.483339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:53.483339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:53.483339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:53.483339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:53.483339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:53.486023 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 301/609] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:53.495303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036ec85b-9edc-455e-90c6-005731286201 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:53.519233 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b045d6d9-f88c-4f43-99f7-2acf599bb6e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:53.519846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b045d6d9-f88c-4f43-99f7-2acf599bb6e7 None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:53.520123 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b045d6d9-f88c-4f43-99f7-2acf599bb6e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:53.520394 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b045d6d9-f88c-4f43-99f7-2acf599bb6e7 None admin] PATH_INFO: `/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/77aac66a6ab44fc1a80ff2fd1572254b/roles/a7b16bb097f14f78a7a31c04bd6d32bd` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:53.524980 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:53.550843 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b045d6d9-f88c-4f43-99f7-2acf599bb6e7 None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4816e95144485e8135e5b8f68c3662, user_id=77aac66a6ab44fc1a80ff2fd1572254b, role_id=a7b16bb097f14f78a7a31c04bd6d32bd)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:53.552337 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b045d6d9-f88c-4f43-99f7-2acf599bb6e7 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:53.576863 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 309/610] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:47:53 2022] PUT /identity/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/77aac66a6ab44fc1a80ff2fd1572254b/roles/a7b16bb097f14f78a7a31c04bd6d32bd => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:47:53.654454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d4a0a9-05bd-44b9-b655-9262a8713956 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:53.654854 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d4a0a9-05bd-44b9-b655-9262a8713956 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:53.655201 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d4a0a9-05bd-44b9-b655-9262a8713956 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:53.656593 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 310/611] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:47:53.663278 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff38ea16-9ce9-4cff-a17b-248b008a5648 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:53.663614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff38ea16-9ce9-4cff-a17b-248b008a5648 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:53.663842 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff38ea16-9ce9-4cff-a17b-248b008a5648 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:53.673544 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036ec85b-9edc-455e-90c6-005731286201 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:53.687297 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-ff38ea16-9ce9-4cff-a17b-248b008a5648 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:53.688738 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-ff38ea16-9ce9-4cff-a17b-248b008a5648 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:53.731435 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:53.770182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc909f6f-e60e-49c8-b0f9-2be1ad83b52e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:53.770724 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc909f6f-e60e-49c8-b0f9-2be1ad83b52e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:53.771003 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc909f6f-e60e-49c8-b0f9-2be1ad83b52e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:53.771274 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc909f6f-e60e-49c8-b0f9-2be1ad83b52e None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:53.772205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc909f6f-e60e-49c8-b0f9-2be1ad83b52e None admin] RBAC: Adding query filter params (name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:53.772508 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc909f6f-e60e-49c8-b0f9-2be1ad83b52e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:53.778803 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc909f6f-e60e-49c8-b0f9-2be1ad83b52e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:53.781826 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 311/612] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:53.789158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 302/613] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:47:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:53.795316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff38ea16-9ce9-4cff-a17b-248b008a5648 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:53.919525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff38ea16-9ce9-4cff-a17b-248b008a5648 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:53.955801 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037ac7ff-ba6f-4853-92c5-9abdf79d13ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:53.956444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037ac7ff-ba6f-4853-92c5-9abdf79d13ba None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:53.956723 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037ac7ff-ba6f-4853-92c5-9abdf79d13ba None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:53.956997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037ac7ff-ba6f-4853-92c5-9abdf79d13ba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:53.974858 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037ac7ff-ba6f-4853-92c5-9abdf79d13ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:53.976370 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037ac7ff-ba6f-4853-92c5-9abdf79d13ba None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:53.996722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-037ac7ff-ba6f-4853-92c5-9abdf79d13ba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:53.996722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:53.996722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:53.996722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:53.996722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:53.996722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:53.996722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:53.996722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:53.996722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:53.996722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:53.996722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:53.996722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:53.996722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:53.996722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:53.996722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:53.996722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:53.998362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:53.999696 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:53.999696 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:53.999696 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:54.002146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 312/614] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:54.003994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc909f6f-e60e-49c8-b0f9-2be1ad83b52e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:54.069321 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037ac7ff-ba6f-4853-92c5-9abdf79d13ba None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:54.108281 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:54.116391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:54.166139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc909f6f-e60e-49c8-b0f9-2be1ad83b52e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:54.212928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccde1212-e382-4183-a5b8-e4d697693947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:54.214280 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccde1212-e382-4183-a5b8-e4d697693947 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:54.215639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccde1212-e382-4183-a5b8-e4d697693947 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:54.216128 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccde1212-e382-4183-a5b8-e4d697693947 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:54.218736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccde1212-e382-4183-a5b8-e4d697693947 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:54.220826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccde1212-e382-4183-a5b8-e4d697693947 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:54.223034 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037ac7ff-ba6f-4853-92c5-9abdf79d13ba None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:54.223746 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccde1212-e382-4183-a5b8-e4d697693947 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:54.252052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 303/615] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:54.255624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccde1212-e382-4183-a5b8-e4d697693947 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:54.300631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c926447-f52a-49c9-a192-1ddb0169bb18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:54.301282 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c926447-f52a-49c9-a192-1ddb0169bb18 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:54.301562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c926447-f52a-49c9-a192-1ddb0169bb18 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:54.301835 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c926447-f52a-49c9-a192-1ddb0169bb18 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:54.325646 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c926447-f52a-49c9-a192-1ddb0169bb18 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:54.332579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-1c926447-f52a-49c9-a192-1ddb0169bb18 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:54.332579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:54.332579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:54.332579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:54.332579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:54.332579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:54.332579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:54.332579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:54.332579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:54.332579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:54.332579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:54.332579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:54.332579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:54.332579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:54.332579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:54.332579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:54.333550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:54.333550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:54.333550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:54.333550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:54.333550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:54.333550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:54.333550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:54.333550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:54.333550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:54.333550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:54.333550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:54.333550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:54.333550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:54.333550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:54.333550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:54.333550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:54.335055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:54.335055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:54.335055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:54.335055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:54.335055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:54.335055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:54.335055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:54.335055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:54.335055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:54.335055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:54.335055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:54.335055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:54.335055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 313/616] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:54.339349 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c926447-f52a-49c9-a192-1ddb0169bb18 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:54.421121 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccde1212-e382-4183-a5b8-e4d697693947 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:54.464171 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfea9846-50d0-4b60-ab39-24c12a83f37b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:54.465742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfea9846-50d0-4b60-ab39-24c12a83f37b None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:54.466023 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfea9846-50d0-4b60-ab39-24c12a83f37b None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:54.466324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfea9846-50d0-4b60-ab39-24c12a83f37b None admin] PATH_INFO: `/v3/users/swift` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:54.471580 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:54.485467 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfea9846-50d0-4b60-ab39-24c12a83f37b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:54.486979 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfea9846-50d0-4b60-ab39-24c12a83f37b None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:54.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-cfea9846-50d0-4b60-ab39-24c12a83f37b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 07 03:47:54.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:54.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:54.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:54.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:54.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:54.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:54.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:54.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:54.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:54.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:54.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:54.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:47:54.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:47:54.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:47:54.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:54.512720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:54.514923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:54.514923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:54.514923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:47:54.514923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:47:54.514923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:47:54.514923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:47:54.514923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:47:54.514923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:47:54.514923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 07 03:47:54.514923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:54.521367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 304/617] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:47:54 2022] GET /identity/v3/users/swift => generated 83 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:54.522237 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfea9846-50d0-4b60-ab39-24c12a83f37b None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:54.539127 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:54.577058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c926447-f52a-49c9-a192-1ddb0169bb18 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:54.629820 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47eaf2f2-1aba-4fe7-837e-6042259bcbdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:54.630467 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47eaf2f2-1aba-4fe7-837e-6042259bcbdd None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:54.630750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47eaf2f2-1aba-4fe7-837e-6042259bcbdd None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:54.631029 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47eaf2f2-1aba-4fe7-837e-6042259bcbdd None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:54.631739 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47eaf2f2-1aba-4fe7-837e-6042259bcbdd None admin] RBAC: Adding query filter params (name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:54.632014 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47eaf2f2-1aba-4fe7-837e-6042259bcbdd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:54.633456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47eaf2f2-1aba-4fe7-837e-6042259bcbdd None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:54.650722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 314/618] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:47:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:54.657785 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47eaf2f2-1aba-4fe7-837e-6042259bcbdd None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:54.677356 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfea9846-50d0-4b60-ab39-24c12a83f37b None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:54.721098 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c530e997-96a0-4746-96b8-d235961d1f8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:54.721699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c530e997-96a0-4746-96b8-d235961d1f8b None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:54.721976 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c530e997-96a0-4746-96b8-d235961d1f8b None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:54.722283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c530e997-96a0-4746-96b8-d235961d1f8b None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:54.723019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c530e997-96a0-4746-96b8-d235961d1f8b None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:54.723320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c530e997-96a0-4746-96b8-d235961d1f8b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:54.724733 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c530e997-96a0-4746-96b8-d235961d1f8b None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:54.743195 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 305/619] 10.222.0.27 () {56 vars in 1135 bytes} [Wed Sep 7 03:47:54 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:54.748900 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c530e997-96a0-4746-96b8-d235961d1f8b None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:54.806531 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:54.845045 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47eaf2f2-1aba-4fe7-837e-6042259bcbdd None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:54.861077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c530e997-96a0-4746-96b8-d235961d1f8b None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:54.919419 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece3eef7-aa7c-4993-9153-57d4aca6fdfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:54.920178 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece3eef7-aa7c-4993-9153-57d4aca6fdfb None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:54.920458 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece3eef7-aa7c-4993-9153-57d4aca6fdfb None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:54.920732 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece3eef7-aa7c-4993-9153-57d4aca6fdfb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:54.923725 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8efc8d-9511-4ffe-8540-467a9d48c981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:54.924350 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8efc8d-9511-4ffe-8540-467a9d48c981 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:54.924619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8efc8d-9511-4ffe-8540-467a9d48c981 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:54.924884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8efc8d-9511-4ffe-8540-467a9d48c981 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:54.943756 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece3eef7-aa7c-4993-9153-57d4aca6fdfb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:54.945216 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece3eef7-aa7c-4993-9153-57d4aca6fdfb None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:54.946372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8efc8d-9511-4ffe-8540-467a9d48c981 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:54.947804 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8efc8d-9511-4ffe-8540-467a9d48c981 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:54.951672 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:54.972221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-ece3eef7-aa7c-4993-9153-57d4aca6fdfb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:54.972221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:54.972221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:54.972221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:54.972221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:54.972221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:54.972221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:54.972221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:54.972221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:54.972221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:54.972221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:54.972221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:54.972221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:54.972221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:54.972221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:54.972221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:54.973227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:54.974829 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:54.974829 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:54.974829 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:54.974829 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 315/620] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:47:54.976100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-aa8efc8d-9511-4ffe-8540-467a9d48c981 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:54.976100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:54.976100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:54.976100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:54.976100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:54.976100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:54.976100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:54.976100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:54.976100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:54.976100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:54.976100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:54.976100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:54.976100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:54.976100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:54.976100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:54.976100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:54.977010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:54.978384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:54.978384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:54.978384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:54.979031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece3eef7-aa7c-4993-9153-57d4aca6fdfb None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:54.980642 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 306/621] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:54.991031 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8efc8d-9511-4ffe-8540-467a9d48c981 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:55.142461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8efc8d-9511-4ffe-8540-467a9d48c981 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:55.154597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:55.171336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece3eef7-aa7c-4993-9153-57d4aca6fdfb None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:55.207567 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4a416d-2dff-4d19-bc96-5d3129e63bba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:55.208339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4a416d-2dff-4d19-bc96-5d3129e63bba None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:55.208755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4a416d-2dff-4d19-bc96-5d3129e63bba None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:55.209129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4a416d-2dff-4d19-bc96-5d3129e63bba None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:55.210167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4a416d-2dff-4d19-bc96-5d3129e63bba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:55.211529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4a416d-2dff-4d19-bc96-5d3129e63bba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:55.213171 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4a416d-2dff-4d19-bc96-5d3129e63bba None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:55.242251 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6239a659-20fd-4138-afbe-c81d8f0ef775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:55.243031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6239a659-20fd-4138-afbe-c81d8f0ef775 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:55.243807 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6239a659-20fd-4138-afbe-c81d8f0ef775 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:55.243999 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 307/622] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:55.245035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4a416d-2dff-4d19-bc96-5d3129e63bba None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:55.245422 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6239a659-20fd-4138-afbe-c81d8f0ef775 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:55.246233 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6239a659-20fd-4138-afbe-c81d8f0ef775 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:55.247012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6239a659-20fd-4138-afbe-c81d8f0ef775 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:55.249159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6239a659-20fd-4138-afbe-c81d8f0ef775 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:55.280779 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 316/623] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:55.286332 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6239a659-20fd-4138-afbe-c81d8f0ef775 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:55.366783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:55.458627 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6239a659-20fd-4138-afbe-c81d8f0ef775 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:55.460909 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4a416d-2dff-4d19-bc96-5d3129e63bba None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:55.503328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f01a44-bd12-4f0f-a977-8c4f82f9629d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:55.505111 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f01a44-bd12-4f0f-a977-8c4f82f9629d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:55.505395 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f01a44-bd12-4f0f-a977-8c4f82f9629d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:55.505670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f01a44-bd12-4f0f-a977-8c4f82f9629d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:55.522090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f01a44-bd12-4f0f-a977-8c4f82f9629d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:55.524523 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f01a44-bd12-4f0f-a977-8c4f82f9629d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:55.541360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9b3d2e-8471-4a07-beb0-9af008bd15cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:55.542004 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9b3d2e-8471-4a07-beb0-9af008bd15cc None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:55.542350 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9b3d2e-8471-4a07-beb0-9af008bd15cc None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:55.542639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9b3d2e-8471-4a07-beb0-9af008bd15cc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:55.551491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-d4f01a44-bd12-4f0f-a977-8c4f82f9629d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:47:55.551491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:55.551491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:55.551491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:55.551491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:55.551491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:55.551491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:55.551491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:55.551491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:55.551491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:55.551491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:55.551491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:55.551491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:47:55.551491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:47:55.551491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:47:55.551491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:55.552490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:55.553454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:55.553454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:55.553454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:47:55.553454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:47:55.553454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:47:55.553454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:47:55.553454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:47:55.553454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:47:55.553454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:47:55.553454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:55.553998 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9b3d2e-8471-4a07-beb0-9af008bd15cc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:55.554078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 317/624] 10.222.0.27 () {56 vars in 1093 bytes} [Wed Sep 7 03:47:55 2022] GET /identity/v3/users/nova => generated 82 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:55.555868 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab9ff73-f6f4-4c5d-9574-2fc6dfdc8631 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:55.555868 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab9ff73-f6f4-4c5d-9574-2fc6dfdc8631 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:55.555868 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab9ff73-f6f4-4c5d-9574-2fc6dfdc8631 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:55.562213 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 318/625] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:55.562371 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9b3d2e-8471-4a07-beb0-9af008bd15cc None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:55.563821 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab9ff73-f6f4-4c5d-9574-2fc6dfdc8631 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:55.574058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-8d9b3d2e-8471-4a07-beb0-9af008bd15cc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:55.574058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:55.574058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:55.574058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:55.574058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:55.574058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:55.574058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:55.574058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:55.574058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:55.574058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:55.574058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:55.574058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:55.574058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:47:55.574058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:47:55.574058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:55.575156 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:55.576120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:47:55.576120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:47:55.576120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:47:55.576120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:47:55.576120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:47:55.576120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:47:55.576120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:55.576120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:55.576120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 308/626] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 331 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:55.576977 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cd3d0f-1443-4c8b-ac62-67c5e8fad2db None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:55.577256 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cd3d0f-1443-4c8b-ac62-67c5e8fad2db None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:55.577529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cd3d0f-1443-4c8b-ac62-67c5e8fad2db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:55.596718 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-86cd3d0f-1443-4c8b-ac62-67c5e8fad2db None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:55.597919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-86cd3d0f-1443-4c8b-ac62-67c5e8fad2db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:55.616410 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:55.669460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 309/627] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:55.670813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cd3d0f-1443-4c8b-ac62-67c5e8fad2db None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:55.689802 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab9ff73-f6f4-4c5d-9574-2fc6dfdc8631 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:55.715953 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:55.735645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf5face-08d2-42f9-bfe1-125c3595b1d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:55.736305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf5face-08d2-42f9-bfe1-125c3595b1d4 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:55.736586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf5face-08d2-42f9-bfe1-125c3595b1d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:55.736862 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf5face-08d2-42f9-bfe1-125c3595b1d4 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:55.737618 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf5face-08d2-42f9-bfe1-125c3595b1d4 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:55.737925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf5face-08d2-42f9-bfe1-125c3595b1d4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:55.740388 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf5face-08d2-42f9-bfe1-125c3595b1d4 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:55.779320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 319/628] 10.222.0.27 () {56 vars in 1133 bytes} [Wed Sep 7 03:47:55 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:55.785097 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf5face-08d2-42f9-bfe1-125c3595b1d4 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:55.820407 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cd3d0f-1443-4c8b-ac62-67c5e8fad2db None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:55.875349 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d34f97-652a-46ad-be79-3a32ebce75ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:55.876070 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d34f97-652a-46ad-be79-3a32ebce75ed None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:55.876353 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d34f97-652a-46ad-be79-3a32ebce75ed None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:55.876667 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d34f97-652a-46ad-be79-3a32ebce75ed None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:55.877400 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d34f97-652a-46ad-be79-3a32ebce75ed None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:55.877833 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d34f97-652a-46ad-be79-3a32ebce75ed None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:55.879446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d34f97-652a-46ad-be79-3a32ebce75ed None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:55.895711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 310/629] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:47:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:55.897093 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42abb3d1-9dad-4977-9381-77c85cae16cc None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:55.897377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42abb3d1-9dad-4977-9381-77c85cae16cc None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:55.897712 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42abb3d1-9dad-4977-9381-77c85cae16cc None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:55.899210 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 311/630] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:55.901508 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42abb3d1-9dad-4977-9381-77c85cae16cc None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:55.919450 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:55.959557 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf5face-08d2-42f9-bfe1-125c3595b1d4 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:55.997192 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b23be6c-f0eb-4f56-b5d1-ce6f76090c40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:55.997936 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b23be6c-f0eb-4f56-b5d1-ce6f76090c40 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:55.998456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b23be6c-f0eb-4f56-b5d1-ce6f76090c40 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:55.998680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b23be6c-f0eb-4f56-b5d1-ce6f76090c40 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:56.018916 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42abb3d1-9dad-4977-9381-77c85cae16cc None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:56.041641 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:56.056729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c9c31c-0d6f-4ca2-8ba9-6c090e6bb345 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:56.057507 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c9c31c-0d6f-4ca2-8ba9-6c090e6bb345 None admin] REQUEST_METHOD: `PUT` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:56.057891 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c9c31c-0d6f-4ca2-8ba9-6c090e6bb345 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:56.058187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c9c31c-0d6f-4ca2-8ba9-6c090e6bb345 None admin] PATH_INFO: `/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/f5c3ab9728f74cd8bb2614bcca9e8619/roles/a7b16bb097f14f78a7a31c04bd6d32bd` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:56.067169 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b23be6c-f0eb-4f56-b5d1-ce6f76090c40 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:56.068790 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b23be6c-f0eb-4f56-b5d1-ce6f76090c40 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:56.073791 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c9c31c-0d6f-4ca2-8ba9-6c090e6bb345 None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4816e95144485e8135e5b8f68c3662, user_id=f5c3ab9728f74cd8bb2614bcca9e8619, role_id=a7b16bb097f14f78a7a31c04bd6d32bd)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:56.075226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c9c31c-0d6f-4ca2-8ba9-6c090e6bb345 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:56.094234 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 312/631] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:47:55 2022] PUT /identity/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/f5c3ab9728f74cd8bb2614bcca9e8619/roles/a7b16bb097f14f78a7a31c04bd6d32bd => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:47:56.095697 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d773b5d5-025c-41dc-bcb4-46f77bb40d6e None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:56.095984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d773b5d5-025c-41dc-bcb4-46f77bb40d6e None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:56.096261 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d773b5d5-025c-41dc-bcb4-46f77bb40d6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:56.120612 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-d773b5d5-025c-41dc-bcb4-46f77bb40d6e None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:56.121950 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-d773b5d5-025c-41dc-bcb4-46f77bb40d6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:56.209417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 320/632] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 07 03:47:56.211753 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b23be6c-f0eb-4f56-b5d1-ce6f76090c40 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:56.213210 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:56.238451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 313/633] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:56.239953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d773b5d5-025c-41dc-bcb4-46f77bb40d6e None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:56.313795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:56.340710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b23be6c-f0eb-4f56-b5d1-ce6f76090c40 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:56.383478 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3f33bf-3cd0-4282-8b9c-60a98ca5c320 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:56.384224 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3f33bf-3cd0-4282-8b9c-60a98ca5c320 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:56.384589 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3f33bf-3cd0-4282-8b9c-60a98ca5c320 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:56.384930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3f33bf-3cd0-4282-8b9c-60a98ca5c320 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:56.386599 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d773b5d5-025c-41dc-bcb4-46f77bb40d6e None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:56.396446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3f33bf-3cd0-4282-8b9c-60a98ca5c320 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:56.401429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-2a3f33bf-3cd0-4282-8b9c-60a98ca5c320 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:56.401429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:56.401429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:56.401429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:56.401429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:56.401429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:56.401429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:56.401429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:56.401429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:56.401429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:56.401429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:56.401429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:56.401429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:56.401429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:56.401429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:56.401429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:56.402624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:56.402624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:56.402624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:56.402624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:56.402624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:56.402624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:56.402624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:56.402624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:56.402624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:56.402624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:56.402624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:56.402624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:56.402624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:56.402624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:56.402624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:56.402624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:56.403671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:56.403671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:56.403671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:56.403671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:56.403671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:56.403671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:56.403671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:56.403671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:56.403671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:56.403671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:56.403671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:56.403671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:56.404686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 321/634] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:56.405938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8458ab3f-c51d-424d-9287-26a0ca265dfc None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:56.406250 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8458ab3f-c51d-424d-9287-26a0ca265dfc None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:56.406526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8458ab3f-c51d-424d-9287-26a0ca265dfc None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:56.407772 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 322/635] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:56.412613 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8458ab3f-c51d-424d-9287-26a0ca265dfc None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:56.433807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc22c369-4798-4fdf-a0b2-a79c29ff4861 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:56.434806 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc22c369-4798-4fdf-a0b2-a79c29ff4861 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:56.435881 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc22c369-4798-4fdf-a0b2-a79c29ff4861 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:56.436887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc22c369-4798-4fdf-a0b2-a79c29ff4861 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:56.452647 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc22c369-4798-4fdf-a0b2-a79c29ff4861 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:56.454776 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc22c369-4798-4fdf-a0b2-a79c29ff4861 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:56.475810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-dc22c369-4798-4fdf-a0b2-a79c29ff4861 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:56.475810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:56.475810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:56.475810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:56.475810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:56.475810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:56.475810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:56.475810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:56.475810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:56.475810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:56.475810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:56.475810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:56.475810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:56.475810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:56.475810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:56.475810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:56.476813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:56.477721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:56.477721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:56.477721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:56.477721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 314/636] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:56.481542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfa5097-2d3c-4cb3-9649-6629ba8c302a None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:56.481542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfa5097-2d3c-4cb3-9649-6629ba8c302a None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:56.481542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfa5097-2d3c-4cb3-9649-6629ba8c302a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:56.499588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-2bfa5097-2d3c-4cb3-9649-6629ba8c302a None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:56.500921 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-2bfa5097-2d3c-4cb3-9649-6629ba8c302a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:56.524666 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8458ab3f-c51d-424d-9287-26a0ca265dfc None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:56.550526 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:56.569791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5243d61-b602-4138-aaa7-2a4f2ba1ddda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:56.570583 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5243d61-b602-4138-aaa7-2a4f2ba1ddda None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:56.570868 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5243d61-b602-4138-aaa7-2a4f2ba1ddda None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:56.571436 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5243d61-b602-4138-aaa7-2a4f2ba1ddda None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:56.572384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5243d61-b602-4138-aaa7-2a4f2ba1ddda None admin] RBAC: Adding query filter params (name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:56.572693 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5243d61-b602-4138-aaa7-2a4f2ba1ddda None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:56.574070 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5243d61-b602-4138-aaa7-2a4f2ba1ddda None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:56.575600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:56.577812 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 315/637] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:56.579916 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfa5097-2d3c-4cb3-9649-6629ba8c302a None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:56.595820 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 323/638] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:47:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:56.596780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5243d61-b602-4138-aaa7-2a4f2ba1ddda None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:56.710711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfa5097-2d3c-4cb3-9649-6629ba8c302a None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:56.715970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5243d61-b602-4138-aaa7-2a4f2ba1ddda None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:56.755817 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1d1ee1-2742-49b6-b97f-a090980f39a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:56.756050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68daac62-6e8d-42b1-8394-457863869c4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:56.756478 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1d1ee1-2742-49b6-b97f-a090980f39a4 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:56.756755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1d1ee1-2742-49b6-b97f-a090980f39a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:56.756903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68daac62-6e8d-42b1-8394-457863869c4c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:56.757004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1d1ee1-2742-49b6-b97f-a090980f39a4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:56.757294 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68daac62-6e8d-42b1-8394-457863869c4c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:56.757741 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68daac62-6e8d-42b1-8394-457863869c4c None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:56.758440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68daac62-6e8d-42b1-8394-457863869c4c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:56.758848 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68daac62-6e8d-42b1-8394-457863869c4c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:56.760304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68daac62-6e8d-42b1-8394-457863869c4c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:56.768521 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1d1ee1-2742-49b6-b97f-a090980f39a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:56.773340 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 316/639] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:56.774434 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68daac62-6e8d-42b1-8394-457863869c4c None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:56.778329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-5a1d1ee1-2742-49b6-b97f-a090980f39a4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:56.778329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:56.778329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:56.778329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:56.778329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:56.778329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:56.778329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:56.778329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:56.778329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:56.778329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:56.778329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:56.778329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:56.778329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:56.778329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:56.778329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:56.778329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:56.779334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:56.779334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:56.779334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:56.779334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:56.779334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:56.779334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:56.779334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:56.779334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:56.779334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:56.779334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:56.779334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:56.779334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:56.779334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:56.779334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:56.779334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:56.779334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:56.780248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:56.780248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:56.780248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:56.780248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:56.780248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:56.780248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:56.780248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:56.780248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:56.780248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:56.780248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:56.780248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:56.780248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:56.780248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 324/640] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:56.780936 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1d1ee1-2742-49b6-b97f-a090980f39a4 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:56.817811 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:56.843301 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:56.889200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1d1ee1-2742-49b6-b97f-a090980f39a4 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:56.892376 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68daac62-6e8d-42b1-8394-457863869c4c None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:56.924618 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0e0e8c-1ef1-4a19-b08a-ec715722aae9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:56.925626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0e0e8c-1ef1-4a19-b08a-ec715722aae9 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:56.926054 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0e0e8c-1ef1-4a19-b08a-ec715722aae9 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:56.926516 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0e0e8c-1ef1-4a19-b08a-ec715722aae9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:56.928674 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daee13b-8c71-44f1-a2f0-b39c1553ffca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:56.929292 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daee13b-8c71-44f1-a2f0-b39c1553ffca None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:56.929570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daee13b-8c71-44f1-a2f0-b39c1553ffca None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:56.929841 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daee13b-8c71-44f1-a2f0-b39c1553ffca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:56.934809 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0e0e8c-1ef1-4a19-b08a-ec715722aae9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:56.936563 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0e0e8c-1ef1-4a19-b08a-ec715722aae9 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:56.940609 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9daee13b-8c71-44f1-a2f0-b39c1553ffca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:56.942033 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9daee13b-8c71-44f1-a2f0-b39c1553ffca None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:56.950212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-1e0e0e8c-1ef1-4a19-b08a-ec715722aae9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:56.950212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:56.950212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:56.950212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:56.950212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:56.950212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:56.950212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:56.950212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:56.950212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:56.950212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:56.950212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:56.950212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:56.950212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:47:56.950212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:47:56.950212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:56.951336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:56.952420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:47:56.952420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:47:56.952420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:47:56.952420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:47:56.952420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:47:56.952420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:47:56.952420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:56.952420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:56.954495 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 325/641] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:56.955433 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0e0e8c-1ef1-4a19-b08a-ec715722aae9 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:56.958625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-9daee13b-8c71-44f1-a2f0-b39c1553ffca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:56.958625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:56.958625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:56.958625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:56.958625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:56.958625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:56.958625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:56.958625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:56.958625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:56.958625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:56.958625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:56.958625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:56.958625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:56.958625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:56.958625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:56.958625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:56.959619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:56.960524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:56.960524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:56.960524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:56.960524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 317/642] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:56.960959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daee13b-8c71-44f1-a2f0-b39c1553ffca None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:57.052659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:57.066606 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0e0e8c-1ef1-4a19-b08a-ec715722aae9 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:57.067454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daee13b-8c71-44f1-a2f0-b39c1553ffca None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:57.086088 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:57.104866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcd8515-911d-4085-88bf-57e81644d8ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:57.105490 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcd8515-911d-4085-88bf-57e81644d8ce None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:57.105764 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcd8515-911d-4085-88bf-57e81644d8ce None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:57.106034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcd8515-911d-4085-88bf-57e81644d8ce None admin] PATH_INFO: `/v3/roles/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:57.108186 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc1d6db-210f-4945-b4d0-79169ebda951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:57.108992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc1d6db-210f-4945-b4d0-79169ebda951 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:57.109426 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc1d6db-210f-4945-b4d0-79169ebda951 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:57.109830 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc1d6db-210f-4945-b4d0-79169ebda951 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:57.110665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc1d6db-210f-4945-b4d0-79169ebda951 None admin] RBAC: Adding query filter params (name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:57.111130 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc1d6db-210f-4945-b4d0-79169ebda951 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:57.112875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc1d6db-210f-4945-b4d0-79169ebda951 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:57.116409 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dcd8515-911d-4085-88bf-57e81644d8ce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:57.121633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-3dcd8515-911d-4085-88bf-57e81644d8ce None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:57.121633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:57.121633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:57.121633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:57.121633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:57.121633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:57.121633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:57.121633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:57.121633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:57.121633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:57.121633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:57.121633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:57.121633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:57.121633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:57.121633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:57.121633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:57.128485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:57.128485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:57.128485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:57.128485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:57.128485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:57.128485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:57.128485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:57.128485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:57.128485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:57.128485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:57.128485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:57.128485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:57.128485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:57.128485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:57.128485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:57.128485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:57.129379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:57.129379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:57.129379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:57.129379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:57.129379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:57.129379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:57.129379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:57.129379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:57.129379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:57.129379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:57.129379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:57.129379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:57.129379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 318/643] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:57.129379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcd8515-911d-4085-88bf-57e81644d8ce None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:57.130307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 326/644] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:47:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:57.130307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc1d6db-210f-4945-b4d0-79169ebda951 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:57.229671 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcd8515-911d-4085-88bf-57e81644d8ce None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:57.235232 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc1d6db-210f-4945-b4d0-79169ebda951 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:57.269855 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fa0e8b-6f57-4cdf-b95e-6b7092551889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:57.270493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fa0e8b-6f57-4cdf-b95e-6b7092551889 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:57.270772 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fa0e8b-6f57-4cdf-b95e-6b7092551889 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:57.271074 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fa0e8b-6f57-4cdf-b95e-6b7092551889 None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:57.271725 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fa0e8b-6f57-4cdf-b95e-6b7092551889 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:57.272029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fa0e8b-6f57-4cdf-b95e-6b7092551889 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:57.272209 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322b506f-0201-47ca-a939-e7e0e09ba63f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:57.272766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322b506f-0201-47ca-a939-e7e0e09ba63f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:57.273102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322b506f-0201-47ca-a939-e7e0e09ba63f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:57.273377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322b506f-0201-47ca-a939-e7e0e09ba63f None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:57.273481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fa0e8b-6f57-4cdf-b95e-6b7092551889 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:57.274012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322b506f-0201-47ca-a939-e7e0e09ba63f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:57.274340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322b506f-0201-47ca-a939-e7e0e09ba63f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:57.275679 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322b506f-0201-47ca-a939-e7e0e09ba63f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:57.292377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 319/645] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:47:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:57.292377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fa0e8b-6f57-4cdf-b95e-6b7092551889 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:57.292864 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 327/646] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:57.292864 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322b506f-0201-47ca-a939-e7e0e09ba63f None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:57.304710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:57.335791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:57.403646 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322b506f-0201-47ca-a939-e7e0e09ba63f None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:57.407228 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fa0e8b-6f57-4cdf-b95e-6b7092551889 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:57.441472 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9df04b0-cb58-4bf9-9766-3a5827362fbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:57.442216 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9df04b0-cb58-4bf9-9766-3a5827362fbb None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:57.442568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9df04b0-cb58-4bf9-9766-3a5827362fbb None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:57.442916 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9df04b0-cb58-4bf9-9766-3a5827362fbb None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:57.443010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f66717-962d-406d-8b18-d9126f4b8638 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:57.443620 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f66717-962d-406d-8b18-d9126f4b8638 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:57.443713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9df04b0-cb58-4bf9-9766-3a5827362fbb None admin] RBAC: Adding query filter params (name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:57.443906 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f66717-962d-406d-8b18-d9126f4b8638 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:57.444113 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9df04b0-cb58-4bf9-9766-3a5827362fbb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:57.444204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f66717-962d-406d-8b18-d9126f4b8638 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:57.445573 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9df04b0-cb58-4bf9-9766-3a5827362fbb None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:57.454875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 328/647] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:47:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:57.455947 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9df04b0-cb58-4bf9-9766-3a5827362fbb None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:57.460193 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f66717-962d-406d-8b18-d9126f4b8638 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:57.461654 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f66717-962d-406d-8b18-d9126f4b8638 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:57.477911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-03f66717-962d-406d-8b18-d9126f4b8638 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:57.477911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:57.477911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:57.477911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:57.477911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:57.477911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:57.477911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:57.477911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:57.477911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:57.477911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:57.477911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:57.477911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:57.477911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:57.477911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:57.477911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:57.477911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:57.479150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:57.480863 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:57.480863 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:57.480863 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:57.484448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 320/648] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:57.485044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f66717-962d-406d-8b18-d9126f4b8638 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:57.548143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:57.561892 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:57.564869 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9df04b0-cb58-4bf9-9766-3a5827362fbb None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:57.594733 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f66717-962d-406d-8b18-d9126f4b8638 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:57.602479 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb737ce-83dd-4ac8-81be-0905cb87d000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:57.603091 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb737ce-83dd-4ac8-81be-0905cb87d000 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:57.603371 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb737ce-83dd-4ac8-81be-0905cb87d000 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:57.603645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb737ce-83dd-4ac8-81be-0905cb87d000 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:57.604373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb737ce-83dd-4ac8-81be-0905cb87d000 None admin] RBAC: Adding query filter params (role.id=a7b16bb097f14f78a7a31c04bd6d32bd, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=953cb6c24bcc4551b91ce540d18cd751) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:57.604681 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb737ce-83dd-4ac8-81be-0905cb87d000 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:57.606106 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb737ce-83dd-4ac8-81be-0905cb87d000 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:57.612043 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 329/649] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:47:57 2022] GET /identity/v3/role_assignments?user.id=953cb6c24bcc4551b91ce540d18cd751&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=a7b16bb097f14f78a7a31c04bd6d32bd => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:57.612820 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb737ce-83dd-4ac8-81be-0905cb87d000 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:57.645879 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95300bf-6906-40ff-b448-b219336e10b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:57.646928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95300bf-6906-40ff-b448-b219336e10b5 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:57.647527 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95300bf-6906-40ff-b448-b219336e10b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:57.647901 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95300bf-6906-40ff-b448-b219336e10b5 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:57.657423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95300bf-6906-40ff-b448-b219336e10b5 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:57.659117 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95300bf-6906-40ff-b448-b219336e10b5 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:57.672391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-d95300bf-6906-40ff-b448-b219336e10b5 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 07 03:47:57.672391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:57.672391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:57.672391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:57.672391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:57.672391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:57.672391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:57.672391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:57.672391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:57.672391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:57.672391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:57.672391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:57.672391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:47:57.672391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:47:57.672391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:47:57.672391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:57.673439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:57.679789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:57.679789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:57.679789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:47:57.679789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:47:57.679789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:47:57.679789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:47:57.679789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:47:57.679789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:47:57.679789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 07 03:47:57.679789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:57.679789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 321/650] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:57 2022] GET /identity/v3/users/cinder => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:57.679789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95300bf-6906-40ff-b448-b219336e10b5 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:57.756072 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb737ce-83dd-4ac8-81be-0905cb87d000 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:57.799074 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95300bf-6906-40ff-b448-b219336e10b5 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:57.807850 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f6cb3f-ff71-4de6-b024-e766ef088e20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:57.809232 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f6cb3f-ff71-4de6-b024-e766ef088e20 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:57.809576 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f6cb3f-ff71-4de6-b024-e766ef088e20 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:57.809922 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f6cb3f-ff71-4de6-b024-e766ef088e20 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:57.825805 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f6cb3f-ff71-4de6-b024-e766ef088e20 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:57.829987 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f6cb3f-ff71-4de6-b024-e766ef088e20 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:57.831938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:57.846090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-16f6cb3f-ff71-4de6-b024-e766ef088e20 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:57.846090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:57.846090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:57.846090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:57.846090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:57.846090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:57.846090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:57.846090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:57.846090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:57.846090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:57.846090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:57.846090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:57.846090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:57.846090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:57.846090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:57.846090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:57.847315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:57.848435 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:57.848435 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:57.848435 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:57.848625 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 330/651] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:57.850545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f6cb3f-ff71-4de6-b024-e766ef088e20 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:57.861011 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fa28ce-63ef-43e7-aa77-ac1f174e1dca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:57.861730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fa28ce-63ef-43e7-aa77-ac1f174e1dca None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:57.862081 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fa28ce-63ef-43e7-aa77-ac1f174e1dca None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:57.862470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fa28ce-63ef-43e7-aa77-ac1f174e1dca None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:57.863200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fa28ce-63ef-43e7-aa77-ac1f174e1dca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:57.863748 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fa28ce-63ef-43e7-aa77-ac1f174e1dca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:57.865331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fa28ce-63ef-43e7-aa77-ac1f174e1dca None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:57.877387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:57.880289 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 322/652] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:57.881252 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fa28ce-63ef-43e7-aa77-ac1f174e1dca None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:57.981078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f6cb3f-ff71-4de6-b024-e766ef088e20 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:58.011851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fa28ce-63ef-43e7-aa77-ac1f174e1dca None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:58.021381 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa3082a-3feb-4b50-9cb1-b661b02e4b5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:58.022458 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa3082a-3feb-4b50-9cb1-b661b02e4b5c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:58.022968 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa3082a-3feb-4b50-9cb1-b661b02e4b5c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:58.023547 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa3082a-3feb-4b50-9cb1-b661b02e4b5c None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:58.024597 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa3082a-3feb-4b50-9cb1-b661b02e4b5c None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:58.025132 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa3082a-3feb-4b50-9cb1-b661b02e4b5c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:58.026912 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa3082a-3feb-4b50-9cb1-b661b02e4b5c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:58.054161 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 331/653] 10.222.0.27 () {56 vars in 1137 bytes} [Wed Sep 7 03:47:57 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:58.054161 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa3082a-3feb-4b50-9cb1-b661b02e4b5c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:58.057972 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5560e061-a875-40bd-b8c5-5f855e34282b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:58.058852 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5560e061-a875-40bd-b8c5-5f855e34282b None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:58.059332 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5560e061-a875-40bd-b8c5-5f855e34282b None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:58.059806 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5560e061-a875-40bd-b8c5-5f855e34282b None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:58.061103 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5560e061-a875-40bd-b8c5-5f855e34282b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:58.061634 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5560e061-a875-40bd-b8c5-5f855e34282b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:58.063237 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5560e061-a875-40bd-b8c5-5f855e34282b None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:58.075603 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 323/654] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:58.077542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5560e061-a875-40bd-b8c5-5f855e34282b None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:58.129876 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:58.141449 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:58.171081 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa3082a-3feb-4b50-9cb1-b661b02e4b5c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:58.202601 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5560e061-a875-40bd-b8c5-5f855e34282b None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:58.213159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48c75b2-94d5-4f16-8a36-160559e9af77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:58.213667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48c75b2-94d5-4f16-8a36-160559e9af77 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:58.213952 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48c75b2-94d5-4f16-8a36-160559e9af77 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:58.214275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48c75b2-94d5-4f16-8a36-160559e9af77 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:58.222197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48c75b2-94d5-4f16-8a36-160559e9af77 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:58.223671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48c75b2-94d5-4f16-8a36-160559e9af77 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:58.241764 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703aa5f5-1931-41c1-9212-01b5fbd33e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:58.242345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-d48c75b2-94d5-4f16-8a36-160559e9af77 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 07 03:47:58.242345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:58.242345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:58.242345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:58.242345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:58.242345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:58.242345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:58.242345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:58.242345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:58.242345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:58.242345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:58.242345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:58.242345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:47:58.242345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:47:58.242345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:47:58.242345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:58.243215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:58.244151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:58.244151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:58.244151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:47:58.244151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:47:58.244151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:47:58.244151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:47:58.244151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:47:58.244151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:47:58.244151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 07 03:47:58.244151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:58.247463 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703aa5f5-1931-41c1-9212-01b5fbd33e29 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:58.248063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703aa5f5-1931-41c1-9212-01b5fbd33e29 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:58.248403 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 332/655] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:47:58 2022] GET /identity/v3/users/glance => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:58.248777 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703aa5f5-1931-41c1-9212-01b5fbd33e29 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:58.249372 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48c75b2-94d5-4f16-8a36-160559e9af77 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:58.328562 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703aa5f5-1931-41c1-9212-01b5fbd33e29 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:58.330718 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703aa5f5-1931-41c1-9212-01b5fbd33e29 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:58.365714 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48c75b2-94d5-4f16-8a36-160559e9af77 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:58.375499 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:58.402311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8123729-2fa0-46bf-b7f1-e66beb35c66c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:58.402959 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8123729-2fa0-46bf-b7f1-e66beb35c66c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:58.403239 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8123729-2fa0-46bf-b7f1-e66beb35c66c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:58.403514 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8123729-2fa0-46bf-b7f1-e66beb35c66c None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:58.406209 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:58.413568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8123729-2fa0-46bf-b7f1-e66beb35c66c None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:58.415474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8123729-2fa0-46bf-b7f1-e66beb35c66c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:58.427121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-a8123729-2fa0-46bf-b7f1-e66beb35c66c None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 07 03:47:58.427121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:58.427121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:58.427121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:58.427121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:58.427121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:58.427121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:58.427121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:58.427121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:58.427121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:58.427121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:58.427121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:58.427121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:47:58.427121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:47:58.427121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:47:58.427121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:58.428058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:58.429026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:58.429026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:58.429026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:47:58.429026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:47:58.429026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:47:58.429026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:47:58.429026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:47:58.429026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:47:58.429026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 07 03:47:58.429026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:58.429026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 333/656] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:58 2022] GET /identity/v3/users/neutron => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:58.429637 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8123729-2fa0-46bf-b7f1-e66beb35c66c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:58.446140 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 324/657] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:58.446679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43638ce-6b36-4ad7-8dff-1e1464c52b98 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:58.449474 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43638ce-6b36-4ad7-8dff-1e1464c52b98 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:58.449474 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43638ce-6b36-4ad7-8dff-1e1464c52b98 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:58.449474 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 325/658] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:47:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:47:58.449474 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43638ce-6b36-4ad7-8dff-1e1464c52b98 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:58.548173 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8123729-2fa0-46bf-b7f1-e66beb35c66c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:58.558423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43638ce-6b36-4ad7-8dff-1e1464c52b98 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:58.585857 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a11e128-9717-44b9-b4ce-70d5e8e55667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:58.586871 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a11e128-9717-44b9-b4ce-70d5e8e55667 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:58.587390 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a11e128-9717-44b9-b4ce-70d5e8e55667 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:58.587889 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a11e128-9717-44b9-b4ce-70d5e8e55667 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:58.588856 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a11e128-9717-44b9-b4ce-70d5e8e55667 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:58.589426 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a11e128-9717-44b9-b4ce-70d5e8e55667 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:58.591050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a11e128-9717-44b9-b4ce-70d5e8e55667 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:58.592978 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7178b2-4f76-4ca2-bbf1-1a7503b9ecc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:58.593577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7178b2-4f76-4ca2-bbf1-1a7503b9ecc0 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:58.593851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7178b2-4f76-4ca2-bbf1-1a7503b9ecc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:58.594132 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7178b2-4f76-4ca2-bbf1-1a7503b9ecc0 None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:58.594866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7178b2-4f76-4ca2-bbf1-1a7503b9ecc0 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:58.595169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7178b2-4f76-4ca2-bbf1-1a7503b9ecc0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:58.596571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7178b2-4f76-4ca2-bbf1-1a7503b9ecc0 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:58.605890 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:58.617673 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 334/659] 10.222.0.27 () {56 vars in 1137 bytes} [Wed Sep 7 03:47:58 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:58.617673 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a11e128-9717-44b9-b4ce-70d5e8e55667 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:58.622202 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 326/660] 10.222.0.27 () {56 vars in 1139 bytes} [Wed Sep 7 03:47:58 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:58.623904 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e697e3d1-37ff-4692-a02e-56a739219125 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:58.624270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e697e3d1-37ff-4692-a02e-56a739219125 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:58.624545 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e697e3d1-37ff-4692-a02e-56a739219125 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:58.643605 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-e697e3d1-37ff-4692-a02e-56a739219125 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:47:58.644770 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-e697e3d1-37ff-4692-a02e-56a739219125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:47:58.673893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:58.718385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 327/661] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:47:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:47:58.719249 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e697e3d1-37ff-4692-a02e-56a739219125 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:58.728899 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a11e128-9717-44b9-b4ce-70d5e8e55667 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:58.769209 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ace9afb-bead-43d6-a03b-74a3a2721972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:58.770237 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ace9afb-bead-43d6-a03b-74a3a2721972 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:58.770713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ace9afb-bead-43d6-a03b-74a3a2721972 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:58.771180 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ace9afb-bead-43d6-a03b-74a3a2721972 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:58.783131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ace9afb-bead-43d6-a03b-74a3a2721972 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:58.784966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ace9afb-bead-43d6-a03b-74a3a2721972 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:58.800945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-6ace9afb-bead-43d6-a03b-74a3a2721972 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:58.800945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:58.800945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:58.800945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:58.800945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:58.800945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:58.800945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:58.800945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:58.800945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:58.800945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:58.800945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:58.800945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:58.800945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:58.800945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:58.800945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:58.800945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:58.802084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:58.803084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:58.803084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:58.803084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:58.807417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 335/662] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:58.808495 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ace9afb-bead-43d6-a03b-74a3a2721972 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:58.839511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e697e3d1-37ff-4692-a02e-56a739219125 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:58.868701 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:58.880145 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133fd243-7f75-4e1b-a31f-12296046a22a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:58.880991 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133fd243-7f75-4e1b-a31f-12296046a22a None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:58.881456 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133fd243-7f75-4e1b-a31f-12296046a22a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:58.882183 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133fd243-7f75-4e1b-a31f-12296046a22a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:58.922349 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ace9afb-bead-43d6-a03b-74a3a2721972 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:58.925651 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:58.962431 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151d9fef-2860-45f9-9697-5c246907bfbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:58.963167 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151d9fef-2860-45f9-9697-5c246907bfbb None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:58.963449 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151d9fef-2860-45f9-9697-5c246907bfbb None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:58.963840 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151d9fef-2860-45f9-9697-5c246907bfbb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:58.974716 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133fd243-7f75-4e1b-a31f-12296046a22a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:58.977516 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133fd243-7f75-4e1b-a31f-12296046a22a None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:59.038070 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151d9fef-2860-45f9-9697-5c246907bfbb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:59.039577 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151d9fef-2860-45f9-9697-5c246907bfbb None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:59.095213 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 328/663] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:59.096464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133fd243-7f75-4e1b-a31f-12296046a22a None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:59.125420 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:59.155457 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 336/664] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:47:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:47:59.156315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151d9fef-2860-45f9-9697-5c246907bfbb None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:59.160495 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:59.208693 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133fd243-7f75-4e1b-a31f-12296046a22a None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:59.244002 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba22015-d65f-4195-8d65-45346bf43c24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:59.244621 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba22015-d65f-4195-8d65-45346bf43c24 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:59.244868 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba22015-d65f-4195-8d65-45346bf43c24 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:59.245139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba22015-d65f-4195-8d65-45346bf43c24 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:59.245775 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba22015-d65f-4195-8d65-45346bf43c24 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:59.246078 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba22015-d65f-4195-8d65-45346bf43c24 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:59.247466 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba22015-d65f-4195-8d65-45346bf43c24 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:59.264864 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 329/665] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:59.265782 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba22015-d65f-4195-8d65-45346bf43c24 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:59.273336 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151d9fef-2860-45f9-9697-5c246907bfbb None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:59.309443 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed9ab79-3a8c-4495-9163-94083fd2a3d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:59.310785 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed9ab79-3a8c-4495-9163-94083fd2a3d9 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:59.310973 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed9ab79-3a8c-4495-9163-94083fd2a3d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:59.311199 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed9ab79-3a8c-4495-9163-94083fd2a3d9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:59.321270 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed9ab79-3a8c-4495-9163-94083fd2a3d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:47:59.326473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-bed9ab79-3a8c-4495-9163-94083fd2a3d9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:59.326473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:59.326473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:59.326473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:59.326473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:59.326473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:59.326473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:59.326473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:59.326473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:59.326473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:59.326473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:59.326473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:59.326473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:47:59.326473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:47:59.326473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:47:59.326473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:47:59.327624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:47:59.327624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:47:59.327624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:47:59.327624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:47:59.327624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:59.327624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:59.327624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:59.327624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:59.327624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:59.327624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:59.327624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:59.327624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:59.327624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:59.327624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:59.327624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:59.327624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:59.328894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:59.328894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:59.328894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:59.328894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:59.328894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:47:59.328894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:47:59.328894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:47:59.328894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:47:59.328894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:47:59.328894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:47:59.328894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:47:59.328894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:59.332565 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 337/666] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:47:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:59.333631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed9ab79-3a8c-4495-9163-94083fd2a3d9 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:59.364732 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:59.377351 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba22015-d65f-4195-8d65-45346bf43c24 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:59.405205 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:59.414643 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ac677b-1ecd-4d39-9058-0d1f20753c75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:59.415383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ac677b-1ecd-4d39-9058-0d1f20753c75 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:59.415670 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ac677b-1ecd-4d39-9058-0d1f20753c75 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:59.415944 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ac677b-1ecd-4d39-9058-0d1f20753c75 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:59.426694 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ac677b-1ecd-4d39-9058-0d1f20753c75 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:59.428126 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ac677b-1ecd-4d39-9058-0d1f20753c75 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:59.443664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-44ac677b-1ecd-4d39-9058-0d1f20753c75 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:59.443664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:59.443664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:59.443664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:59.443664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:59.443664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:59.443664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:59.443664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:59.443664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:59.443664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:59.443664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:59.443664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:59.443664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:59.443664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:59.443664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:59.443664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:59.444617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:59.445501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:59.445501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:59.445501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:59.445501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 330/667] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:59.445720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ac677b-1ecd-4d39-9058-0d1f20753c75 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:59.451803 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed9ab79-3a8c-4495-9163-94083fd2a3d9 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:59.489393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabb9710-4ded-4f9a-b6db-cab041881529 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:59.490060 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabb9710-4ded-4f9a-b6db-cab041881529 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:59.490351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabb9710-4ded-4f9a-b6db-cab041881529 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:59.490626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabb9710-4ded-4f9a-b6db-cab041881529 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:59.503717 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabb9710-4ded-4f9a-b6db-cab041881529 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:59.506025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabb9710-4ded-4f9a-b6db-cab041881529 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:59.529083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-aabb9710-4ded-4f9a-b6db-cab041881529 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:59.529083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:59.529083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:59.529083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:59.529083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:59.529083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:59.529083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:59.529083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:59.529083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:59.529083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:59.529083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:59.529083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:59.529083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:47:59.529083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:47:59.529083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:47:59.529083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:59.530271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:47:59.531166 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:47:59.531166 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:47:59.531166 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:47:59.533947 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 338/668] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:59.535416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabb9710-4ded-4f9a-b6db-cab041881529 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:59.562846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ac677b-1ecd-4d39-9058-0d1f20753c75 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:59.601036 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc41892a-8af9-451e-afc0-8b196c676540 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:59.602105 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc41892a-8af9-451e-afc0-8b196c676540 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:59.602504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc41892a-8af9-451e-afc0-8b196c676540 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:59.602893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc41892a-8af9-451e-afc0-8b196c676540 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:59.616774 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc41892a-8af9-451e-afc0-8b196c676540 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:59.618558 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc41892a-8af9-451e-afc0-8b196c676540 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:59.623733 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:59.633710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-bc41892a-8af9-451e-afc0-8b196c676540 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:59.633710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:47:59.633710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:47:59.633710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:47:59.633710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:47:59.633710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:47:59.633710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:47:59.633710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:47:59.633710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:47:59.633710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:47:59.633710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:47:59.633710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:47:59.633710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:47:59.633710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:47:59.633710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:47:59.634859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:47:59.635917 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:47:59.635917 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:47:59.635917 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:47:59.635917 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:47:59.635917 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:47:59.635917 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:47:59.635917 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:47:59.635917 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:47:59.640232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 331/669] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:47:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:47:59.641137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc41892a-8af9-451e-afc0-8b196c676540 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:59.678906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabb9710-4ded-4f9a-b6db-cab041881529 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:59.699739 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:59.724662 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a19ec5d-4753-44de-89db-a0fb5f2e9a77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:59.725471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a19ec5d-4753-44de-89db-a0fb5f2e9a77 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:59.725775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a19ec5d-4753-44de-89db-a0fb5f2e9a77 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:59.726053 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a19ec5d-4753-44de-89db-a0fb5f2e9a77 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:59.726747 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a19ec5d-4753-44de-89db-a0fb5f2e9a77 None admin] RBAC: Adding query filter params (name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:59.727058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a19ec5d-4753-44de-89db-a0fb5f2e9a77 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:59.728448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a19ec5d-4753-44de-89db-a0fb5f2e9a77 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:59.739867 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 339/670] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:47:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:59.743121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a19ec5d-4753-44de-89db-a0fb5f2e9a77 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:59.775384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc41892a-8af9-451e-afc0-8b196c676540 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:59.818292 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e313ac44-65d2-45da-a4ca-cc896119b2ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:59.819167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e313ac44-65d2-45da-a4ca-cc896119b2ef None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:59.819451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e313ac44-65d2-45da-a4ca-cc896119b2ef None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:59.819727 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e313ac44-65d2-45da-a4ca-cc896119b2ef None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:59.820379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e313ac44-65d2-45da-a4ca-cc896119b2ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:59.820688 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e313ac44-65d2-45da-a4ca-cc896119b2ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:59.822293 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e313ac44-65d2-45da-a4ca-cc896119b2ef None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:59.843361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 332/671] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:59.844391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e313ac44-65d2-45da-a4ca-cc896119b2ef None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:59.880508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a19ec5d-4753-44de-89db-a0fb5f2e9a77 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:59.926062 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46289fb-b785-4f2a-9dc5-6e42b537f62a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:47:59.929105 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46289fb-b785-4f2a-9dc5-6e42b537f62a None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:47:59.929518 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46289fb-b785-4f2a-9dc5-6e42b537f62a None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:47:59.929942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46289fb-b785-4f2a-9dc5-6e42b537f62a None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:47:59.933035 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46289fb-b785-4f2a-9dc5-6e42b537f62a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:47:59.933350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46289fb-b785-4f2a-9dc5-6e42b537f62a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:47:59.935198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46289fb-b785-4f2a-9dc5-6e42b537f62a None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:47:59.941701 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:47:59.955946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 340/672] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:47:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:47:59.955946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46289fb-b785-4f2a-9dc5-6e42b537f62a None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:47:59.983720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e313ac44-65d2-45da-a4ca-cc896119b2ef None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:47:59.993555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:00.018749 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86891255-0894-4532-94fb-4526f9c40c69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:00.019465 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86891255-0894-4532-94fb-4526f9c40c69 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:00.019827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86891255-0894-4532-94fb-4526f9c40c69 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:00.020172 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86891255-0894-4532-94fb-4526f9c40c69 None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:00.020937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86891255-0894-4532-94fb-4526f9c40c69 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:00.021244 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86891255-0894-4532-94fb-4526f9c40c69 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:00.022721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86891255-0894-4532-94fb-4526f9c40c69 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:00.034664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 333/673] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:47:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:00.035579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86891255-0894-4532-94fb-4526f9c40c69 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:00.054746 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46289fb-b785-4f2a-9dc5-6e42b537f62a None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:00.088651 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8f2e96-edd8-490e-9aaf-066a2088eebc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:00.089279 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8f2e96-edd8-490e-9aaf-066a2088eebc None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:00.089561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8f2e96-edd8-490e-9aaf-066a2088eebc None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:00.089837 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8f2e96-edd8-490e-9aaf-066a2088eebc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:00.100426 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8f2e96-edd8-490e-9aaf-066a2088eebc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:00.101881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8f2e96-edd8-490e-9aaf-066a2088eebc None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:00.116674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-db8f2e96-edd8-490e-9aaf-066a2088eebc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:00.116674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:00.116674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:00.116674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:00.116674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:00.116674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:00.116674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:00.116674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:00.116674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:00.116674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:00.116674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:00.116674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:00.116674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:00.116674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:00.116674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:00.116674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:00.117776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:00.118915 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:00.118915 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:00.118915 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:00.118915 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 341/674] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:47:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:00.119272 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492fef26-39d3-465a-9f66-6142b955d8e5 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:00.119534 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492fef26-39d3-465a-9f66-6142b955d8e5 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:00.119815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492fef26-39d3-465a-9f66-6142b955d8e5 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:00.121172 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 342/675] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:00 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:00.123428 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492fef26-39d3-465a-9f66-6142b955d8e5 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:00.139459 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86891255-0894-4532-94fb-4526f9c40c69 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:00.159647 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:00.174722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0414b36-3f27-4fd5-8536-972133383d0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:00.175358 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0414b36-3f27-4fd5-8536-972133383d0e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:00.175643 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0414b36-3f27-4fd5-8536-972133383d0e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:00.175918 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0414b36-3f27-4fd5-8536-972133383d0e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:00.182965 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0414b36-3f27-4fd5-8536-972133383d0e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:00.184402 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0414b36-3f27-4fd5-8536-972133383d0e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:00.196048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-b0414b36-3f27-4fd5-8536-972133383d0e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:00.196048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:00.196048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:00.196048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:00.196048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:00.196048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:00.196048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:00.196048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:00.196048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:00.196048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:00.196048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:00.196048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:00.196048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:00.196048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:00.196048 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:00.197089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:00.197966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:00.197966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:00.197966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:00.197966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:00.197966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:00.197966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:00.197966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:00.197966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:00.197966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 334/676] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:00.197966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0414b36-3f27-4fd5-8536-972133383d0e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:00.219128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:00.230194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492fef26-39d3-465a-9f66-6142b955d8e5 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:00.265603 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0850bf4-2462-41ab-8417-72f4636c5a0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:00.266274 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0850bf4-2462-41ab-8417-72f4636c5a0d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:00.266564 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0850bf4-2462-41ab-8417-72f4636c5a0d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:00.266840 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0850bf4-2462-41ab-8417-72f4636c5a0d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:00.273567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0850bf4-2462-41ab-8417-72f4636c5a0d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:00.275053 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0850bf4-2462-41ab-8417-72f4636c5a0d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:00.286444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-c0850bf4-2462-41ab-8417-72f4636c5a0d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:00.286444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:00.286444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:00.286444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:00.286444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:00.286444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:00.286444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:00.286444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:00.286444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:00.286444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:00.286444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:00.286444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:00.286444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:00.286444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:00.286444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:00.287483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:00.288550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:00.288550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:00.288550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:00.288550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:00.288550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:00.288550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:00.288550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:00.288550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:00.288550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 343/677] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:00.289143 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0850bf4-2462-41ab-8417-72f4636c5a0d None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:00.304868 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0414b36-3f27-4fd5-8536-972133383d0e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:00.338615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84ef0a9-f84b-4dea-a2b2-baaf7fe93549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:00.339210 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84ef0a9-f84b-4dea-a2b2-baaf7fe93549 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:00.339488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84ef0a9-f84b-4dea-a2b2-baaf7fe93549 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:00.339757 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84ef0a9-f84b-4dea-a2b2-baaf7fe93549 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:00.340471 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84ef0a9-f84b-4dea-a2b2-baaf7fe93549 None admin] RBAC: Adding query filter params (role.id=a7b16bb097f14f78a7a31c04bd6d32bd, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=989081d67c8945dc8d853bb2769c902a) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:00.340776 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84ef0a9-f84b-4dea-a2b2-baaf7fe93549 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:00.342339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84ef0a9-f84b-4dea-a2b2-baaf7fe93549 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:00.348169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 335/678] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:48:00 2022] GET /identity/v3/role_assignments?user.id=989081d67c8945dc8d853bb2769c902a&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=a7b16bb097f14f78a7a31c04bd6d32bd => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:00.349367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a65ffe-c498-49bf-a656-2734921b285a None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:00.349656 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a65ffe-c498-49bf-a656-2734921b285a None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:00.349932 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a65ffe-c498-49bf-a656-2734921b285a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:00.372309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-e0a65ffe-c498-49bf-a656-2734921b285a None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:00.374337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-e0a65ffe-c498-49bf-a656-2734921b285a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:00.402759 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0850bf4-2462-41ab-8417-72f4636c5a0d None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:00.426587 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:00.444763 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55e4c11-0721-4599-87e8-4ce8c9e88787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:00.445591 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55e4c11-0721-4599-87e8-4ce8c9e88787 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:00.445975 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55e4c11-0721-4599-87e8-4ce8c9e88787 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:00.446375 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55e4c11-0721-4599-87e8-4ce8c9e88787 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:00.447161 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55e4c11-0721-4599-87e8-4ce8c9e88787 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:00.447569 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55e4c11-0721-4599-87e8-4ce8c9e88787 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:00.449076 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55e4c11-0721-4599-87e8-4ce8c9e88787 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:00.458211 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 336/679] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:48:00.459382 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a65ffe-c498-49bf-a656-2734921b285a None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:00.470765 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 344/680] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:00.471969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55e4c11-0721-4599-87e8-4ce8c9e88787 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:00.486681 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:00.595249 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55e4c11-0721-4599-87e8-4ce8c9e88787 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:00.607297 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a65ffe-c498-49bf-a656-2734921b285a None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:00.632951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16af012-e2e6-416c-a6e0-7d040146a297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:00.633616 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16af012-e2e6-416c-a6e0-7d040146a297 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:00.633896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16af012-e2e6-416c-a6e0-7d040146a297 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:00.634194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16af012-e2e6-416c-a6e0-7d040146a297 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:00.634887 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16af012-e2e6-416c-a6e0-7d040146a297 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:00.635187 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16af012-e2e6-416c-a6e0-7d040146a297 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:00.636669 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16af012-e2e6-416c-a6e0-7d040146a297 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:00.653613 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 345/681] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:00.653613 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16af012-e2e6-416c-a6e0-7d040146a297 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:00.657702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d60ee5-2cf9-49d8-96a0-e2969ddef1aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:00.658351 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d60ee5-2cf9-49d8-96a0-e2969ddef1aa None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:00.658674 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d60ee5-2cf9-49d8-96a0-e2969ddef1aa None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:00.658945 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d60ee5-2cf9-49d8-96a0-e2969ddef1aa None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:00.660551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d60ee5-2cf9-49d8-96a0-e2969ddef1aa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:00.660861 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d60ee5-2cf9-49d8-96a0-e2969ddef1aa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:00.662305 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d60ee5-2cf9-49d8-96a0-e2969ddef1aa None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:00.677430 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 337/682] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:00.678526 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d60ee5-2cf9-49d8-96a0-e2969ddef1aa None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:00.722455 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:00.751597 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:00.786743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16af012-e2e6-416c-a6e0-7d040146a297 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:00.796277 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d60ee5-2cf9-49d8-96a0-e2969ddef1aa None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:00.822596 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4b9621-0a62-4747-bd84-24e50793a0e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:00.823226 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4b9621-0a62-4747-bd84-24e50793a0e4 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:00.823505 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4b9621-0a62-4747-bd84-24e50793a0e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:00.823775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4b9621-0a62-4747-bd84-24e50793a0e4 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:00.830821 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108aff67-7b37-4741-b056-75f3193f48ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:00.831427 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108aff67-7b37-4741-b056-75f3193f48ff None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:00.831709 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108aff67-7b37-4741-b056-75f3193f48ff None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:00.831982 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108aff67-7b37-4741-b056-75f3193f48ff None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:00.832093 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4b9621-0a62-4747-bd84-24e50793a0e4 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:00.832700 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108aff67-7b37-4741-b056-75f3193f48ff None admin] RBAC: Adding query filter params (role.id=a7b16bb097f14f78a7a31c04bd6d32bd, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=77aac66a6ab44fc1a80ff2fd1572254b) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:00.833009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108aff67-7b37-4741-b056-75f3193f48ff None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:00.833446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4b9621-0a62-4747-bd84-24e50793a0e4 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:00.834477 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108aff67-7b37-4741-b056-75f3193f48ff None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:00.840608 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 338/683] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:48:00 2022] GET /identity/v3/role_assignments?user.id=77aac66a6ab44fc1a80ff2fd1572254b&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=a7b16bb097f14f78a7a31c04bd6d32bd => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:00.842248 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108aff67-7b37-4741-b056-75f3193f48ff None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:00.845417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-bf4b9621-0a62-4747-bd84-24e50793a0e4 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 07 03:48:00.845417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:00.845417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:00.845417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:00.845417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:00.845417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:00.845417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:00.845417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:00.845417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:00.845417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:00.845417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:00.845417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:00.845417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:00.845417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:00.845417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:00.845417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:00.846401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:00.847329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:00.847329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:00.847329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:00.847329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:00.847329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:00.847329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:00.847329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:00.847329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:00.847329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 07 03:48:00.847329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:00.851570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 346/684] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:48:00 2022] GET /identity/v3/users/swift => generated 83 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:00.852559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4b9621-0a62-4747-bd84-24e50793a0e4 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:00.974652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:00.978907 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108aff67-7b37-4741-b056-75f3193f48ff None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:00.986353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4b9621-0a62-4747-bd84-24e50793a0e4 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:01.020380 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cbb0d8-251d-483e-b844-104a574646c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:01.021004 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cbb0d8-251d-483e-b844-104a574646c6 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:01.021284 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cbb0d8-251d-483e-b844-104a574646c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:01.021557 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cbb0d8-251d-483e-b844-104a574646c6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:01.022455 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cbb0d8-251d-483e-b844-104a574646c6 None admin] RBAC: Adding query filter params (role.id=a7b16bb097f14f78a7a31c04bd6d32bd, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=dfa09be69ee3444e94f00cc139bf50e7) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:01.022868 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cbb0d8-251d-483e-b844-104a574646c6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:01.024415 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cbb0d8-251d-483e-b844-104a574646c6 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:01.025193 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:01.032979 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114c6991-3c79-47bc-afcd-9941c292aa85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:01.038440 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114c6991-3c79-47bc-afcd-9941c292aa85 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:01.038440 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114c6991-3c79-47bc-afcd-9941c292aa85 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:01.038440 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114c6991-3c79-47bc-afcd-9941c292aa85 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:01.038748 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 339/685] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:48:00 2022] GET /identity/v3/role_assignments?user.id=dfa09be69ee3444e94f00cc139bf50e7&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=a7b16bb097f14f78a7a31c04bd6d32bd => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:01.041800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cbb0d8-251d-483e-b844-104a574646c6 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:01.047857 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114c6991-3c79-47bc-afcd-9941c292aa85 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:01.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-114c6991-3c79-47bc-afcd-9941c292aa85 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:01.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:01.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:01.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:01.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:01.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:01.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:01.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:01.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:01.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:01.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:01.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:01.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:01.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:01.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:01.055311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:01.056306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:01.056306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:01.056306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:01.056306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:01.056306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:01.056306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:01.056306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:01.056306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:01.056306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:01.056306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:01.056306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:01.056306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:01.056306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:01.056306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:01.056306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:01.056306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:01.057131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:01.057131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:01.057131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:01.057131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:01.057131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:01.057131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:01.057131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:01.057131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:01.057131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:01.057131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:01.057131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:01.057131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:01.057131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 347/686] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:48:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:01.062980 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114c6991-3c79-47bc-afcd-9941c292aa85 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:01.171477 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cbb0d8-251d-483e-b844-104a574646c6 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:01.213589 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114c6991-3c79-47bc-afcd-9941c292aa85 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:01.218744 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596f7b3b-35e8-4b35-827f-6d87d877076e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:01.219366 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596f7b3b-35e8-4b35-827f-6d87d877076e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:01.219641 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596f7b3b-35e8-4b35-827f-6d87d877076e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:01.219912 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596f7b3b-35e8-4b35-827f-6d87d877076e None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:01.220719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596f7b3b-35e8-4b35-827f-6d87d877076e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:01.221027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596f7b3b-35e8-4b35-827f-6d87d877076e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:01.222531 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596f7b3b-35e8-4b35-827f-6d87d877076e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:01.253286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 340/687] 10.222.0.27 () {56 vars in 1135 bytes} [Wed Sep 7 03:48:01 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:01.259038 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596f7b3b-35e8-4b35-827f-6d87d877076e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:01.262653 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a82e824-95a4-46ba-ad96-7e33f1610263 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:01.262653 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a82e824-95a4-46ba-ad96-7e33f1610263 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:01.262653 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a82e824-95a4-46ba-ad96-7e33f1610263 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:01.262915 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a82e824-95a4-46ba-ad96-7e33f1610263 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:01.264357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a82e824-95a4-46ba-ad96-7e33f1610263 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:01.266598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a82e824-95a4-46ba-ad96-7e33f1610263 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:01.266598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a82e824-95a4-46ba-ad96-7e33f1610263 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:01.277747 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 348/688] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:48:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:01.283739 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a82e824-95a4-46ba-ad96-7e33f1610263 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:01.287795 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:01.349895 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:01.374084 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596f7b3b-35e8-4b35-827f-6d87d877076e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:01.403302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a82e824-95a4-46ba-ad96-7e33f1610263 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:01.410333 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c444f04-bd8e-4d72-adf0-06d28295c6d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:01.410999 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c444f04-bd8e-4d72-adf0-06d28295c6d4 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:01.411277 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c444f04-bd8e-4d72-adf0-06d28295c6d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:01.411598 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c444f04-bd8e-4d72-adf0-06d28295c6d4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:01.440567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3151ce-6c9a-4811-8cef-40a3b13449c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:01.441303 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3151ce-6c9a-4811-8cef-40a3b13449c4 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:01.441627 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3151ce-6c9a-4811-8cef-40a3b13449c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:01.441858 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3151ce-6c9a-4811-8cef-40a3b13449c4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:01.455086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3151ce-6c9a-4811-8cef-40a3b13449c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:01.456641 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3151ce-6c9a-4811-8cef-40a3b13449c4 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:01.473434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-ce3151ce-6c9a-4811-8cef-40a3b13449c4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:01.473434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:01.473434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:01.473434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:01.473434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:01.473434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:01.473434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:01.473434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:01.473434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:01.473434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:01.473434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:01.473434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:01.473434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:01.473434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:01.473434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:01.473434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:01.474469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:01.475387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:01.475387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:01.475387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:01.475387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 349/689] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:01.480478 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3151ce-6c9a-4811-8cef-40a3b13449c4 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:01.485733 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c444f04-bd8e-4d72-adf0-06d28295c6d4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:01.487225 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c444f04-bd8e-4d72-adf0-06d28295c6d4 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:01.509303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:01.595829 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3151ce-6c9a-4811-8cef-40a3b13449c4 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:01.599564 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:01.599949 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 341/690] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:48:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 07 03:48:01.601347 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c444f04-bd8e-4d72-adf0-06d28295c6d4 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:01.638321 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f4db80-6853-412b-b958-d3dfe26faf83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:01.639311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f4db80-6853-412b-b958-d3dfe26faf83 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:01.639799 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f4db80-6853-412b-b958-d3dfe26faf83 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:01.640244 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f4db80-6853-412b-b958-d3dfe26faf83 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:01.641063 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f4db80-6853-412b-b958-d3dfe26faf83 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:01.641541 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f4db80-6853-412b-b958-d3dfe26faf83 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:01.643218 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f4db80-6853-412b-b958-d3dfe26faf83 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:01.658378 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 350/691] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:01.664085 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f4db80-6853-412b-b958-d3dfe26faf83 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:01.706166 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c444f04-bd8e-4d72-adf0-06d28295c6d4 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:01.727957 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:01.743257 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff42f4f1-218c-4dff-91f7-b7cc63655889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:01.743901 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff42f4f1-218c-4dff-91f7-b7cc63655889 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:01.744182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff42f4f1-218c-4dff-91f7-b7cc63655889 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:01.744455 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff42f4f1-218c-4dff-91f7-b7cc63655889 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:01.755899 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff42f4f1-218c-4dff-91f7-b7cc63655889 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:01.757759 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff42f4f1-218c-4dff-91f7-b7cc63655889 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:01.775019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-ff42f4f1-218c-4dff-91f7-b7cc63655889 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:01.775019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:01.775019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:01.775019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:01.775019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:01.775019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:01.775019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:01.775019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:01.775019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:01.775019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:01.775019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:01.775019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:01.775019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:01.775019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:01.775019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:01.775019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:01.775953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:01.776829 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:01.776829 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:01.776829 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:01.780610 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f4db80-6853-412b-b958-d3dfe26faf83 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:01.780736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 342/692] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:01.781934 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff42f4f1-218c-4dff-91f7-b7cc63655889 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:01.818354 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2475c4bf-d160-4aa6-9999-3089e29edeab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:01.819393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2475c4bf-d160-4aa6-9999-3089e29edeab None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:01.819949 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2475c4bf-d160-4aa6-9999-3089e29edeab None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:01.820395 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2475c4bf-d160-4aa6-9999-3089e29edeab None admin] PATH_INFO: `/v3/users/nova` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:01.829252 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2475c4bf-d160-4aa6-9999-3089e29edeab None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:01.831104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2475c4bf-d160-4aa6-9999-3089e29edeab None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:01.844090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-2475c4bf-d160-4aa6-9999-3089e29edeab None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:48:01.844090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:01.844090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:01.844090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:01.844090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:01.844090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:01.844090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:01.844090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:01.844090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:01.844090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:01.844090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:01.844090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:01.844090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:01.844090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:01.844090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:01.844090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:01.845036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:01.845938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:01.845938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:01.845938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:01.845938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:01.845938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:01.845938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:01.845938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:01.845938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:01.845938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:48:01.845938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:01.848476 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 351/693] 10.222.0.27 () {56 vars in 1093 bytes} [Wed Sep 7 03:48:01 2022] GET /identity/v3/users/nova => generated 82 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:01.854019 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2475c4bf-d160-4aa6-9999-3089e29edeab None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:01.881844 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:01.892924 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff42f4f1-218c-4dff-91f7-b7cc63655889 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:01.928868 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc46e564-3c8b-47ae-bdd8-1e0e0b699669 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:01.929751 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc46e564-3c8b-47ae-bdd8-1e0e0b699669 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:01.930209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc46e564-3c8b-47ae-bdd8-1e0e0b699669 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:01.930705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc46e564-3c8b-47ae-bdd8-1e0e0b699669 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:01.931498 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc46e564-3c8b-47ae-bdd8-1e0e0b699669 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:01.931992 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc46e564-3c8b-47ae-bdd8-1e0e0b699669 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:01.933527 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc46e564-3c8b-47ae-bdd8-1e0e0b699669 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:01.947657 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 343/694] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:01.962772 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc46e564-3c8b-47ae-bdd8-1e0e0b699669 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:01.962995 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2475c4bf-d160-4aa6-9999-3089e29edeab None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:02.010003 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:02.011747 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b260b1df-b3c7-4e93-95d4-dbdade68e4ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:02.012373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b260b1df-b3c7-4e93-95d4-dbdade68e4ee None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:02.012651 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b260b1df-b3c7-4e93-95d4-dbdade68e4ee None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:02.012925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b260b1df-b3c7-4e93-95d4-dbdade68e4ee None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:02.014294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b260b1df-b3c7-4e93-95d4-dbdade68e4ee None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:02.014606 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b260b1df-b3c7-4e93-95d4-dbdade68e4ee None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:02.016033 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b260b1df-b3c7-4e93-95d4-dbdade68e4ee None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:02.051787 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 352/695] 10.222.0.27 () {56 vars in 1133 bytes} [Wed Sep 7 03:48:01 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:02.063219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b260b1df-b3c7-4e93-95d4-dbdade68e4ee None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:02.165114 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc46e564-3c8b-47ae-bdd8-1e0e0b699669 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:02.198673 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:02.215092 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b260b1df-b3c7-4e93-95d4-dbdade68e4ee None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:02.255215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110b08c3-b650-4514-a05b-17f62ef5b90a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:02.255867 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110b08c3-b650-4514-a05b-17f62ef5b90a None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:02.256145 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110b08c3-b650-4514-a05b-17f62ef5b90a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:02.256423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110b08c3-b650-4514-a05b-17f62ef5b90a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:02.262559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c3559c-edac-4a83-91be-3eaf783e10c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:02.263973 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c3559c-edac-4a83-91be-3eaf783e10c8 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:02.264265 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c3559c-edac-4a83-91be-3eaf783e10c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:02.264625 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c3559c-edac-4a83-91be-3eaf783e10c8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:02.271057 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110b08c3-b650-4514-a05b-17f62ef5b90a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:02.272512 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110b08c3-b650-4514-a05b-17f62ef5b90a None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:02.298046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-110b08c3-b650-4514-a05b-17f62ef5b90a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:02.298046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:02.298046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:02.298046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:02.298046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:02.298046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:02.298046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:02.298046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:02.298046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:02.298046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:02.298046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:02.298046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:02.298046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:02.298046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:02.298046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:02.298995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:02.301089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:02.301089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:02.301089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:02.301089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:02.301089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:02.301089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:02.301089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:02.301089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:02.301089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 344/696] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:02.305173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110b08c3-b650-4514-a05b-17f62ef5b90a None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:02.333883 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c3559c-edac-4a83-91be-3eaf783e10c8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:02.335498 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c3559c-edac-4a83-91be-3eaf783e10c8 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:02.422519 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:02.439550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 353/697] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:48:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 07 03:48:02.448829 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c3559c-edac-4a83-91be-3eaf783e10c8 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:02.512270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:02.531520 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110b08c3-b650-4514-a05b-17f62ef5b90a None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:02.569789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87380d2-ae02-401d-b510-d14bd33cef98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:02.570518 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87380d2-ae02-401d-b510-d14bd33cef98 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:02.570896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87380d2-ae02-401d-b510-d14bd33cef98 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:02.571334 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87380d2-ae02-401d-b510-d14bd33cef98 None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:02.572203 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c87380d2-ae02-401d-b510-d14bd33cef98 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:02.572642 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c87380d2-ae02-401d-b510-d14bd33cef98 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:02.574223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c87380d2-ae02-401d-b510-d14bd33cef98 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:02.588620 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 345/698] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:02.595405 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87380d2-ae02-401d-b510-d14bd33cef98 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:02.619006 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c3559c-edac-4a83-91be-3eaf783e10c8 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:02.655749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba17e4b2-8035-4921-ae77-c7824f7da06e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:02.656389 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba17e4b2-8035-4921-ae77-c7824f7da06e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:02.656671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba17e4b2-8035-4921-ae77-c7824f7da06e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:02.656953 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba17e4b2-8035-4921-ae77-c7824f7da06e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:02.667551 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba17e4b2-8035-4921-ae77-c7824f7da06e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:02.669006 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba17e4b2-8035-4921-ae77-c7824f7da06e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:02.684710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-ba17e4b2-8035-4921-ae77-c7824f7da06e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:02.684710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:02.684710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:02.684710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:02.684710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:02.684710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:02.684710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:02.684710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:02.684710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:02.684710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:02.684710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:02.684710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:02.684710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:02.684710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:02.684710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:02.684710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:02.690162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:02.691284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:02.691284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:02.691284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:02.691284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 354/699] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:02.699146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba17e4b2-8035-4921-ae77-c7824f7da06e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:02.770563 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:02.794212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87380d2-ae02-401d-b510-d14bd33cef98 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:02.832610 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489ead82-e968-4ab7-aca0-9d8fb5903861 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:02.833490 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489ead82-e968-4ab7-aca0-9d8fb5903861 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:02.833986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489ead82-e968-4ab7-aca0-9d8fb5903861 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:02.834460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489ead82-e968-4ab7-aca0-9d8fb5903861 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:02.835388 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489ead82-e968-4ab7-aca0-9d8fb5903861 None admin] RBAC: Adding query filter params (role.id=a7b16bb097f14f78a7a31c04bd6d32bd, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=f5c3ab9728f74cd8bb2614bcca9e8619) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:02.835877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489ead82-e968-4ab7-aca0-9d8fb5903861 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:02.837486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489ead82-e968-4ab7-aca0-9d8fb5903861 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:02.844865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 346/700] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:48:02 2022] GET /identity/v3/role_assignments?user.id=f5c3ab9728f74cd8bb2614bcca9e8619&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=a7b16bb097f14f78a7a31c04bd6d32bd => generated 608 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:02.882870 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba17e4b2-8035-4921-ae77-c7824f7da06e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:02.890492 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7884ab2-4026-46ee-9b93-8c6014fa3b86 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:02.890997 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7884ab2-4026-46ee-9b93-8c6014fa3b86 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:02.891454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7884ab2-4026-46ee-9b93-8c6014fa3b86 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:02.894448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 347/701] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:48:02.899460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9717cbc1-069e-4a36-9884-f74561ba57cd None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:02.900067 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9717cbc1-069e-4a36-9884-f74561ba57cd None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:02.900521 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9717cbc1-069e-4a36-9884-f74561ba57cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:02.925982 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b38996-8f0d-4a92-b52c-a00765cffe02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:02.927759 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b38996-8f0d-4a92-b52c-a00765cffe02 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:02.928042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b38996-8f0d-4a92-b52c-a00765cffe02 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:02.928316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b38996-8f0d-4a92-b52c-a00765cffe02 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:02.929249 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b38996-8f0d-4a92-b52c-a00765cffe02 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:02.929559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b38996-8f0d-4a92-b52c-a00765cffe02 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:02.931267 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b38996-8f0d-4a92-b52c-a00765cffe02 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:02.938197 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-9717cbc1-069e-4a36-9884-f74561ba57cd None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:02.939431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-9717cbc1-069e-4a36-9884-f74561ba57cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:02.959654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 355/702] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:02.963897 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:02.967409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b38996-8f0d-4a92-b52c-a00765cffe02 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:03.074247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 348/703] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:48:03.120107 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:03.128864 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b38996-8f0d-4a92-b52c-a00765cffe02 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:03.164424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8135102-b816-40de-b5e9-51fcf05a8314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:03.165073 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8135102-b816-40de-b5e9-51fcf05a8314 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:03.165357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8135102-b816-40de-b5e9-51fcf05a8314 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:03.165635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8135102-b816-40de-b5e9-51fcf05a8314 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:03.172875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8135102-b816-40de-b5e9-51fcf05a8314 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:03.174445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8135102-b816-40de-b5e9-51fcf05a8314 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:03.187815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-d8135102-b816-40de-b5e9-51fcf05a8314 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:03.187815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:03.187815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:03.187815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:03.187815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:03.187815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:03.187815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:03.187815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:03.187815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:03.187815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:03.187815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:03.187815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:03.187815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:03.187815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:03.187815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:03.188736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:03.189904 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:03.189904 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:03.189904 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:03.189904 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:03.189904 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:03.189904 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:03.189904 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:03.189904 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:03.189904 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 356/704] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:03.197316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9717cbc1-069e-4a36-9884-f74561ba57cd None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:03.309731 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9717cbc1-069e-4a36-9884-f74561ba57cd None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:03.345953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f684289-0731-4078-ab3f-3886b35aadcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:03.346963 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f684289-0731-4078-ab3f-3886b35aadcd None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:03.347446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f684289-0731-4078-ab3f-3886b35aadcd None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:03.347916 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f684289-0731-4078-ab3f-3886b35aadcd None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:03.348835 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f684289-0731-4078-ab3f-3886b35aadcd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:03.349395 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f684289-0731-4078-ab3f-3886b35aadcd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:03.351061 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f684289-0731-4078-ab3f-3886b35aadcd None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:03.367817 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 349/705] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:03.369055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8135102-b816-40de-b5e9-51fcf05a8314 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:03.370831 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f684289-0731-4078-ab3f-3886b35aadcd None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:03.380228 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:03.475684 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f684289-0731-4078-ab3f-3886b35aadcd None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:03.485134 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8135102-b816-40de-b5e9-51fcf05a8314 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:03.511803 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81523eaa-6158-486d-94e3-0302e5163d39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:03.512716 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81523eaa-6158-486d-94e3-0302e5163d39 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:03.514311 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81523eaa-6158-486d-94e3-0302e5163d39 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:03.514797 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81523eaa-6158-486d-94e3-0302e5163d39 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:03.515751 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81523eaa-6158-486d-94e3-0302e5163d39 None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=953cb6c24bcc4551b91ce540d18cd751) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:03.516252 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81523eaa-6158-486d-94e3-0302e5163d39 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:03.517937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81523eaa-6158-486d-94e3-0302e5163d39 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:03.526781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caa5eed-3968-44d7-b9a7-59e289f89d1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:03.526781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caa5eed-3968-44d7-b9a7-59e289f89d1c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:03.527059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caa5eed-3968-44d7-b9a7-59e289f89d1c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:03.527301 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 350/706] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:48:03 2022] GET /identity/v3/role_assignments?user.id=953cb6c24bcc4551b91ce540d18cd751&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 259 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:03.527429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caa5eed-3968-44d7-b9a7-59e289f89d1c None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:03.528044 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5caa5eed-3968-44d7-b9a7-59e289f89d1c None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:03.529710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5caa5eed-3968-44d7-b9a7-59e289f89d1c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-5caa5eed-3968-44d7-b9a7-59e289f89d1c None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:03.543450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:03.545104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:03.545104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:03.545104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:03.545104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:03.545104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:03.545104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:03.545104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:03.545104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:03.545104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:03.545104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:03.545104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:03.545104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:03.545104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:03.545104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:03.545104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:03.545104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:03.546316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:03.546316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:03.546316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:03.546316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:03.546316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:03.546316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:03.546316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:03.546316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:03.546316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:03.546316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:03.546316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:03.546316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:03.546316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 07 03:48:03.546316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:03.546316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 357/707] 10.222.0.27 () {56 vars in 1129 bytes} [Wed Sep 7 03:48:03 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:03.552371 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81523eaa-6158-486d-94e3-0302e5163d39 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:03.635137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:03.664954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d1959f-d494-4c76-aac5-ad6f67b00dd1 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:03.665472 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d1959f-d494-4c76-aac5-ad6f67b00dd1 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:03.665549 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d1959f-d494-4c76-aac5-ad6f67b00dd1 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:03.667171 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 358/708] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:48:03.673145 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a0ca4a-fbeb-4d3e-930c-430b6c2a627c None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:03.673427 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a0ca4a-fbeb-4d3e-930c-430b6c2a627c None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:03.673707 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a0ca4a-fbeb-4d3e-930c-430b6c2a627c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:03.677284 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81523eaa-6158-486d-94e3-0302e5163d39 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:03.707293 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-29a0ca4a-fbeb-4d3e-930c-430b6c2a627c None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:03.708679 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-29a0ca4a-fbeb-4d3e-930c-430b6c2a627c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:03.731616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1be79ba-6c73-483b-945f-a6b8afb16a71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:03.732803 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1be79ba-6c73-483b-945f-a6b8afb16a71 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:03.733263 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1be79ba-6c73-483b-945f-a6b8afb16a71 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:03.733524 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:03.733730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1be79ba-6c73-483b-945f-a6b8afb16a71 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:03.734639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1be79ba-6c73-483b-945f-a6b8afb16a71 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:03.735019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1be79ba-6c73-483b-945f-a6b8afb16a71 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:03.736545 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1be79ba-6c73-483b-945f-a6b8afb16a71 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:03.752116 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 351/709] 10.222.0.27 () {56 vars in 1124 bytes} [Wed Sep 7 03:48:03 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:03.760376 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1be79ba-6c73-483b-945f-a6b8afb16a71 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:03.802835 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 359/710] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:48:03.885720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1be79ba-6c73-483b-945f-a6b8afb16a71 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:03.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b09dc49-3c22-476d-a72e-51a77e59cef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:03.921661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b09dc49-3c22-476d-a72e-51a77e59cef9 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:03.921941 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b09dc49-3c22-476d-a72e-51a77e59cef9 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:03.922247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b09dc49-3c22-476d-a72e-51a77e59cef9 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:03.922886 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b09dc49-3c22-476d-a72e-51a77e59cef9 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:03.923189 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b09dc49-3c22-476d-a72e-51a77e59cef9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:03.924555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b09dc49-3c22-476d-a72e-51a77e59cef9 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:03.925967 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:03.931306 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 352/711] 10.222.0.27 () {56 vars in 1124 bytes} [Wed Sep 7 03:48:03 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:04.116865 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a0ca4a-fbeb-4d3e-930c-430b6c2a627c None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:04.132177 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1d408-f850-4062-986f-8d1d829fce2c None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:04.132625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1d408-f850-4062-986f-8d1d829fce2c None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:04.133024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1d408-f850-4062-986f-8d1d829fce2c None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:04.138661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 353/712] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:48:04.141291 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73c641d-9a3d-43ca-96e0-a1f0b4489a5b None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:04.141679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73c641d-9a3d-43ca-96e0-a1f0b4489a5b None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:04.142210 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73c641d-9a3d-43ca-96e0-a1f0b4489a5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:04.161808 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-e73c641d-9a3d-43ca-96e0-a1f0b4489a5b None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:04.163290 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-e73c641d-9a3d-43ca-96e0-a1f0b4489a5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:04.246260 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 354/713] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:48:04.294264 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a0ca4a-fbeb-4d3e-930c-430b6c2a627c None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:04.331907 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df93b93-1004-415a-b5cd-459039dd185d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:04.332832 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df93b93-1004-415a-b5cd-459039dd185d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:04.333267 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df93b93-1004-415a-b5cd-459039dd185d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:04.333698 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df93b93-1004-415a-b5cd-459039dd185d None admin] PATH_INFO: `/v3/services/network` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:04.334465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df93b93-1004-415a-b5cd-459039dd185d None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:04.335992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df93b93-1004-415a-b5cd-459039dd185d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-9df93b93-1004-415a-b5cd-459039dd185d None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:04.346214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:04.347570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:04.347570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:04.347570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:04.347570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:04.347570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:04.347570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:04.347570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:04.347570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:04.347570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:04.347570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:04.347570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:04.347570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:04.347570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:04.347570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:04.347570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:04.347570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:04.348654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:04.348654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:04.348654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:04.348654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:04.348654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:04.348654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:04.348654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:04.348654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:04.348654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:04.348654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:04.348654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:04.348654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:04.348654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 07 03:48:04.348654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:04.350869 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 360/714] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:04 2022] GET /identity/v3/services/network => generated 88 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:04.356631 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73c641d-9a3d-43ca-96e0-a1f0b4489a5b None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:04.480622 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73c641d-9a3d-43ca-96e0-a1f0b4489a5b None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:04.508678 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:04.517828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1053d6ed-bb9f-4651-99b8-445f129cb3c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:04.518551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1053d6ed-bb9f-4651-99b8-445f129cb3c9 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:04.518802 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1053d6ed-bb9f-4651-99b8-445f129cb3c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:04.519080 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1053d6ed-bb9f-4651-99b8-445f129cb3c9 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:04.519716 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1053d6ed-bb9f-4651-99b8-445f129cb3c9 None admin] RBAC: Adding query filter params (name=network) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:04.520016 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1053d6ed-bb9f-4651-99b8-445f129cb3c9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:04.521384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1053d6ed-bb9f-4651-99b8-445f129cb3c9 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:04.526388 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 355/715] 10.222.0.27 () {56 vars in 1112 bytes} [Wed Sep 7 03:48:04 2022] GET /identity/v3/services?name=network => generated 124 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:04.535403 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df93b93-1004-415a-b5cd-459039dd185d None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:04.564580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:04.681226 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df93b93-1004-415a-b5cd-459039dd185d None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:04.692621 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1053d6ed-bb9f-4651-99b8-445f129cb3c9 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:04.725559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386918ed-4ee4-4954-a9e9-f43b99e772f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:04.726558 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386918ed-4ee4-4954-a9e9-f43b99e772f3 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:04.727238 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386918ed-4ee4-4954-a9e9-f43b99e772f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:04.728912 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386918ed-4ee4-4954-a9e9-f43b99e772f3 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:04.729754 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386918ed-4ee4-4954-a9e9-f43b99e772f3 None admin] RBAC: Adding query filter params (type=network) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:04.731143 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386918ed-4ee4-4954-a9e9-f43b99e772f3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:04.733272 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386918ed-4ee4-4954-a9e9-f43b99e772f3 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:04.745182 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 361/716] 10.222.0.27 () {56 vars in 1112 bytes} [Wed Sep 7 03:48:04 2022] GET /identity/v3/services?type=network => generated 124 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:04.828929 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1053d6ed-bb9f-4651-99b8-445f129cb3c9 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:04.866714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27eb92e2-f8c4-47f8-a5c0-f0ee09b65c27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:04.867752 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eb92e2-f8c4-47f8-a5c0-f0ee09b65c27 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:04.868238 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eb92e2-f8c4-47f8-a5c0-f0ee09b65c27 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:04.868736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eb92e2-f8c4-47f8-a5c0-f0ee09b65c27 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:04.881439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27eb92e2-f8c4-47f8-a5c0-f0ee09b65c27 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:04.883265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27eb92e2-f8c4-47f8-a5c0-f0ee09b65c27 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:04.899594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-27eb92e2-f8c4-47f8-a5c0-f0ee09b65c27 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:04.899594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:04.899594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:04.899594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:04.899594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:04.899594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:04.899594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:04.899594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:04.899594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:04.899594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:04.899594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:04.899594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:04.899594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:04.899594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:04.899594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:04.899594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:04.900602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:04.901486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:04.901486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:04.901486 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:04.903800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 356/717] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:04.910261 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386918ed-4ee4-4954-a9e9-f43b99e772f3 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:04.997691 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:05.024146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386918ed-4ee4-4954-a9e9-f43b99e772f3 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:05.065762 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ccf1d9-5652-462a-be01-550a04c5b599 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:05.066433 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ccf1d9-5652-462a-be01-550a04c5b599 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:05.066718 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ccf1d9-5652-462a-be01-550a04c5b599 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:05.067059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ccf1d9-5652-462a-be01-550a04c5b599 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:05.067767 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ccf1d9-5652-462a-be01-550a04c5b599 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:05.068133 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ccf1d9-5652-462a-be01-550a04c5b599 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:05.069579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ccf1d9-5652-462a-be01-550a04c5b599 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:05.084552 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 362/718] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:05.090208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27eb92e2-f8c4-47f8-a5c0-f0ee09b65c27 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:05.140991 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:05.201799 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27eb92e2-f8c4-47f8-a5c0-f0ee09b65c27 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:05.241739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d697c8c-5e1c-4858-ab64-4a7441eeb4ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:05.242460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d697c8c-5e1c-4858-ab64-4a7441eeb4ae None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:05.242743 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d697c8c-5e1c-4858-ab64-4a7441eeb4ae None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:05.243072 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d697c8c-5e1c-4858-ab64-4a7441eeb4ae None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:05.243741 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d697c8c-5e1c-4858-ab64-4a7441eeb4ae None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:05.245229 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d697c8c-5e1c-4858-ab64-4a7441eeb4ae None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:05.278260 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 357/719] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:48:05 2022] POST /identity/v3/users => generated 254 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:48:06.039500 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb821d9-da49-4545-a73f-19abb11ad8ee None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:06.040020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb821d9-da49-4545-a73f-19abb11ad8ee None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:06.040020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb821d9-da49-4545-a73f-19abb11ad8ee None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:06.041581 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 363/720] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:06.047071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6113ee-085d-414d-aa8b-997d3dcadf96 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:06.047532 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6113ee-085d-414d-aa8b-997d3dcadf96 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:06.047618 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6113ee-085d-414d-aa8b-997d3dcadf96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:06.072822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-ca6113ee-085d-414d-aa8b-997d3dcadf96 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:06.074019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-ca6113ee-085d-414d-aa8b-997d3dcadf96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:06.172670 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 358/721] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:48:06.189089 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1185c41-c350-4ea8-84c1-58d4a6b5b161 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:06.189360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1185c41-c350-4ea8-84c1-58d4a6b5b161 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:06.189638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1185c41-c350-4ea8-84c1-58d4a6b5b161 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:06.191166 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 364/722] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:06.196367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e97ed5-e88a-4ba8-a40a-62e49f74ec4d None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:06.196657 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e97ed5-e88a-4ba8-a40a-62e49f74ec4d None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:06.196933 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e97ed5-e88a-4ba8-a40a-62e49f74ec4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:06.231132 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-74e97ed5-e88a-4ba8-a40a-62e49f74ec4d None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:06.232853 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-74e97ed5-e88a-4ba8-a40a-62e49f74ec4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:06.266698 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:06.328609 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 359/723] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 07 03:48:06.497994 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1185c41-c350-4ea8-84c1-58d4a6b5b161 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:06.631446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1185c41-c350-4ea8-84c1-58d4a6b5b161 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:06.684703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c467692-42b6-4284-b437-4e7b07c7e8cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:06.685889 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c467692-42b6-4284-b437-4e7b07c7e8cf None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:06.686194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c467692-42b6-4284-b437-4e7b07c7e8cf None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:06.686500 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c467692-42b6-4284-b437-4e7b07c7e8cf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:06.695171 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:06.704417 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e97ed5-e88a-4ba8-a40a-62e49f74ec4d None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:06.707188 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c467692-42b6-4284-b437-4e7b07c7e8cf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:06.713896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-6c467692-42b6-4284-b437-4e7b07c7e8cf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:06.713896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:06.713896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:06.713896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:06.713896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:06.713896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:06.713896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:06.713896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:06.713896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:06.713896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:06.713896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:06.713896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:06.713896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:06.713896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:06.713896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:06.713896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:06.714946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:06.714946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:06.714946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:06.714946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:06.714946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:06.714946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:06.714946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:06.714946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:06.714946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:06.714946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:06.714946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:06.714946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:06.714946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:06.714946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:06.714946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:06.714946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:06.716944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:06.716944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:06.716944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:06.716944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:06.716944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:06.716944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:06.716944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:06.716944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:06.716944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:06.716944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:06.716944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:06.716944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:06.716944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 365/724] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:48:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:06.724136 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c467692-42b6-4284-b437-4e7b07c7e8cf None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:06.888705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e97ed5-e88a-4ba8-a40a-62e49f74ec4d None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:06.903345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c467692-42b6-4284-b437-4e7b07c7e8cf None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:06.936348 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd2d61b-eed3-4e9b-aadc-2ae6bd123c54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:06.938020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd2d61b-eed3-4e9b-aadc-2ae6bd123c54 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:06.938560 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd2d61b-eed3-4e9b-aadc-2ae6bd123c54 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:06.939288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd2d61b-eed3-4e9b-aadc-2ae6bd123c54 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:06.945179 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72f5408-40ef-464a-865b-00c2d69c3290 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:06.945880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72f5408-40ef-464a-865b-00c2d69c3290 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:06.946188 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72f5408-40ef-464a-865b-00c2d69c3290 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:06.946463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72f5408-40ef-464a-865b-00c2d69c3290 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:06.947131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72f5408-40ef-464a-865b-00c2d69c3290 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:06.947435 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72f5408-40ef-464a-865b-00c2d69c3290 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:06.948988 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72f5408-40ef-464a-865b-00c2d69c3290 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:06.952714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd2d61b-eed3-4e9b-aadc-2ae6bd123c54 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:06.959470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-dcd2d61b-eed3-4e9b-aadc-2ae6bd123c54 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:06.959470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:06.959470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:06.959470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:06.959470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:06.959470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:06.959470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:06.959470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:06.959470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:06.959470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:06.959470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:06.959470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:06.959470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:06.959470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:06.959470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:06.959470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:06.960408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:06.960408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:06.960408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:06.960408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:06.960408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:06.960408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:06.960408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:06.960408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:06.960408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:06.960408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:06.960408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:06.960408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:06.960408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:06.960408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:06.960408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:06.960408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:06.961615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:06.961615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:06.961615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:06.961615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:06.961615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:06.961615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:06.961615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:06.961615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:06.961615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:06.961615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:06.961615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:06.961615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:06.964160 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 360/725] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:48:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:06.964390 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 366/726] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:48:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:06.965539 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72f5408-40ef-464a-865b-00c2d69c3290 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:06.966602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9aad61c-8f7e-421b-9281-b11e14c78a2f None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:06.967132 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9aad61c-8f7e-421b-9281-b11e14c78a2f None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:06.967623 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9aad61c-8f7e-421b-9281-b11e14c78a2f None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:06.969700 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 361/727] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:06.970647 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9aad61c-8f7e-421b-9281-b11e14c78a2f None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:06.998824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:07.038342 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:07.085364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72f5408-40ef-464a-865b-00c2d69c3290 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:07.088183 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9aad61c-8f7e-421b-9281-b11e14c78a2f None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:07.122653 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10355c3f-b942-41fd-97e0-661ec29f6373 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:07.123663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10355c3f-b942-41fd-97e0-661ec29f6373 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:07.124148 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10355c3f-b942-41fd-97e0-661ec29f6373 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:07.124642 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10355c3f-b942-41fd-97e0-661ec29f6373 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:07.125300 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f183d84e-9a7d-4c56-9bb3-62bfc68de145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:07.125903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f183d84e-9a7d-4c56-9bb3-62bfc68de145 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:07.126204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f183d84e-9a7d-4c56-9bb3-62bfc68de145 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:07.126476 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f183d84e-9a7d-4c56-9bb3-62bfc68de145 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:07.127134 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f183d84e-9a7d-4c56-9bb3-62bfc68de145 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:07.127490 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f183d84e-9a7d-4c56-9bb3-62bfc68de145 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:07.129163 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f183d84e-9a7d-4c56-9bb3-62bfc68de145 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:07.137749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10355c3f-b942-41fd-97e0-661ec29f6373 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:07.138645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 362/728] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:48:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:07.140066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5fa7df-6b89-41ca-b007-62b7f669e0a7 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:07.140295 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5fa7df-6b89-41ca-b007-62b7f669e0a7 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:07.140564 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5fa7df-6b89-41ca-b007-62b7f669e0a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:07.141241 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10355c3f-b942-41fd-97e0-661ec29f6373 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:07.157406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-10355c3f-b942-41fd-97e0-661ec29f6373 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:07.157406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:07.157406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:07.157406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:07.157406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:07.157406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:07.157406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:07.157406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:07.157406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:07.157406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:07.157406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:07.157406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:07.157406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:07.157406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:07.157406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:07.157406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:07.158448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:07.159312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:07.159312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:07.159312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:07.159491 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-0f5fa7df-6b89-41ca-b007-62b7f669e0a7 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:07.160323 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-0f5fa7df-6b89-41ca-b007-62b7f669e0a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:07.161750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 367/729] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:07.162854 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10355c3f-b942-41fd-97e0-661ec29f6373 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:07.239346 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 363/730] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:48:07.246422 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5fa7df-6b89-41ca-b007-62b7f669e0a7 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:07.271074 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10355c3f-b942-41fd-97e0-661ec29f6373 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:07.273985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:07.282885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:07.309457 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1514c842-f972-4184-9c2e-7fe52c38d0f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:07.310339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1514c842-f972-4184-9c2e-7fe52c38d0f5 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:07.310724 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1514c842-f972-4184-9c2e-7fe52c38d0f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:07.311094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1514c842-f972-4184-9c2e-7fe52c38d0f5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:07.322497 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1514c842-f972-4184-9c2e-7fe52c38d0f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:07.324071 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1514c842-f972-4184-9c2e-7fe52c38d0f5 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:07.340144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-1514c842-f972-4184-9c2e-7fe52c38d0f5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:07.340144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:07.340144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:07.340144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:07.340144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:07.340144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:07.340144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:07.340144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:07.340144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:07.340144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:07.340144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:07.340144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:07.340144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:07.340144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:07.340144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:07.340144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:07.341158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:07.342048 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:07.342048 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:07.342048 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:07.342048 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 368/731] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:07.346929 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1514c842-f972-4184-9c2e-7fe52c38d0f5 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:07.461751 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5fa7df-6b89-41ca-b007-62b7f669e0a7 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:07.525452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0bf144-232a-4702-a091-36fcdbbb7fa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:07.526063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0bf144-232a-4702-a091-36fcdbbb7fa8 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:07.526373 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0bf144-232a-4702-a091-36fcdbbb7fa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:07.526644 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0bf144-232a-4702-a091-36fcdbbb7fa8 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:07.527289 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d0bf144-232a-4702-a091-36fcdbbb7fa8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:07.527593 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d0bf144-232a-4702-a091-36fcdbbb7fa8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:07.529623 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d0bf144-232a-4702-a091-36fcdbbb7fa8 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:07.547079 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1514c842-f972-4184-9c2e-7fe52c38d0f5 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:07.555905 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 364/732] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:07.557198 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33155258-aad2-4589-bbca-5ba5e035b247 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:07.557577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33155258-aad2-4589-bbca-5ba5e035b247 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:07.557858 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33155258-aad2-4589-bbca-5ba5e035b247 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:07.564413 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 365/733] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:07.565256 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33155258-aad2-4589-bbca-5ba5e035b247 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:07.584932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54c8101-66eb-409d-8bea-3b64f52902a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:07.585551 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54c8101-66eb-409d-8bea-3b64f52902a9 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:07.585829 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54c8101-66eb-409d-8bea-3b64f52902a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:07.586103 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54c8101-66eb-409d-8bea-3b64f52902a9 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:07.586790 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54c8101-66eb-409d-8bea-3b64f52902a9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:07.587096 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54c8101-66eb-409d-8bea-3b64f52902a9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:07.588489 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54c8101-66eb-409d-8bea-3b64f52902a9 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:07.601485 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 369/734] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:07.602963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5b7199-1072-49fa-bcd7-10c037809a87 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:07.603248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5b7199-1072-49fa-bcd7-10c037809a87 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:07.603524 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5b7199-1072-49fa-bcd7-10c037809a87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:07.623708 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-8c5b7199-1072-49fa-bcd7-10c037809a87 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:07.625176 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-8c5b7199-1072-49fa-bcd7-10c037809a87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:07.646170 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:07.648820 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:07.683251 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33155258-aad2-4589-bbca-5ba5e035b247 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:07.704027 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 370/735] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 07 03:48:07.705066 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5b7199-1072-49fa-bcd7-10c037809a87 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:07.725152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c369d12-ad8f-4161-8038-eebc184882ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:07.725852 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c369d12-ad8f-4161-8038-eebc184882ba None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:07.726259 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c369d12-ad8f-4161-8038-eebc184882ba None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:07.726630 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c369d12-ad8f-4161-8038-eebc184882ba None admin] PATH_INFO: `/v3/users/nova` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:07.736004 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c369d12-ad8f-4161-8038-eebc184882ba None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:07.737566 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c369d12-ad8f-4161-8038-eebc184882ba None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:07.752232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-5c369d12-ad8f-4161-8038-eebc184882ba None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:48:07.752232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:07.752232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:07.752232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:07.752232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:07.752232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:07.752232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:07.752232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:07.752232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:07.752232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:07.752232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:07.752232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:07.752232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:07.752232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:07.752232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:07.752232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:07.753391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:07.754658 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:07.754658 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:07.754658 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:07.754658 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:07.754658 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:07.754658 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:07.754658 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:07.754658 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:07.754658 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:48:07.754658 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:07.760690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 366/736] 10.222.0.27 () {56 vars in 1093 bytes} [Wed Sep 7 03:48:07 2022] GET /identity/v3/users/nova => generated 82 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:07.760690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c369d12-ad8f-4161-8038-eebc184882ba None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:07.838678 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5b7199-1072-49fa-bcd7-10c037809a87 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:07.875627 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600b362d-fbe8-4699-8040-6366e15421dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:07.877128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600b362d-fbe8-4699-8040-6366e15421dd None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:07.877632 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600b362d-fbe8-4699-8040-6366e15421dd None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:07.878323 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600b362d-fbe8-4699-8040-6366e15421dd None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:07.879139 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600b362d-fbe8-4699-8040-6366e15421dd None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:07.881540 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600b362d-fbe8-4699-8040-6366e15421dd None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:07.886202 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c369d12-ad8f-4161-8038-eebc184882ba None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:07.911595 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 371/737] 10.222.0.27 () {60 vars in 1141 bytes} [Wed Sep 7 03:48:07 2022] POST /identity/v3/services => generated 251 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:48:07.911595 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600b362d-fbe8-4699-8040-6366e15421dd None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:07.943686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:07.944454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:07.981435 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0af23ce-3c85-4e0d-b5f0-c1e51f3efedf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:07.982590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0af23ce-3c85-4e0d-b5f0-c1e51f3efedf None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:07.982919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0af23ce-3c85-4e0d-b5f0-c1e51f3efedf None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:07.983190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0af23ce-3c85-4e0d-b5f0-c1e51f3efedf None admin] PATH_INFO: `/v3/users/swift` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:08.001647 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0af23ce-3c85-4e0d-b5f0-c1e51f3efedf None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:08.004828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0af23ce-3c85-4e0d-b5f0-c1e51f3efedf None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:08.021877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-d0af23ce-3c85-4e0d-b5f0-c1e51f3efedf None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 07 03:48:08.021877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:08.021877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:08.021877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:08.021877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:08.021877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:08.021877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:08.021877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:08.021877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:08.021877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:08.021877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:08.021877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:08.021877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:08.021877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:08.021877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:08.021877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:08.022913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:08.023847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:08.023847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:08.023847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:08.023847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:08.023847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:08.023847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:08.023847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:08.023847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:08.023847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 07 03:48:08.023847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:08.023847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 367/738] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:48:07 2022] GET /identity/v3/users/swift => generated 83 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:08.024432 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0af23ce-3c85-4e0d-b5f0-c1e51f3efedf None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:08.083646 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600b362d-fbe8-4699-8040-6366e15421dd None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:08.128267 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5249058f-f5aa-4f83-ace6-a8043e8d4610 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:08.128958 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5249058f-f5aa-4f83-ace6-a8043e8d4610 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:08.129171 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5249058f-f5aa-4f83-ace6-a8043e8d4610 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:08.129443 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5249058f-f5aa-4f83-ace6-a8043e8d4610 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:08.130397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5249058f-f5aa-4f83-ace6-a8043e8d4610 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:08.130873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5249058f-f5aa-4f83-ace6-a8043e8d4610 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:08.132499 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5249058f-f5aa-4f83-ace6-a8043e8d4610 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:08.142338 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0af23ce-3c85-4e0d-b5f0-c1e51f3efedf None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:08.152151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 372/739] 10.222.0.27 () {56 vars in 1133 bytes} [Wed Sep 7 03:48:07 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:08.153179 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5249058f-f5aa-4f83-ace6-a8043e8d4610 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:08.183002 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb98c14e-1f20-4c3f-acac-7ba40621fc49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:08.183890 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb98c14e-1f20-4c3f-acac-7ba40621fc49 None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:08.184337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb98c14e-1f20-4c3f-acac-7ba40621fc49 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:08.184615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb98c14e-1f20-4c3f-acac-7ba40621fc49 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:08.185807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb98c14e-1f20-4c3f-acac-7ba40621fc49 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:08.187360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb98c14e-1f20-4c3f-acac-7ba40621fc49 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:08.205032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 368/740] 10.222.0.27 () {60 vars in 1141 bytes} [Wed Sep 7 03:48:08 2022] POST /identity/v3/services => generated 240 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:48:08.206789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f64a15-acca-4056-9973-a681c5e5b2fc None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:08.207084 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f64a15-acca-4056-9973-a681c5e5b2fc None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:08.207362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f64a15-acca-4056-9973-a681c5e5b2fc None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:08.208577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 369/741] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:08.209521 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f64a15-acca-4056-9973-a681c5e5b2fc None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:08.257739 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:08.287357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:08.300867 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5249058f-f5aa-4f83-ace6-a8043e8d4610 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:08.358571 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df401a33-6a17-4e96-8940-e8bb1f7e52e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:08.359484 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df401a33-6a17-4e96-8940-e8bb1f7e52e9 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:08.359914 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df401a33-6a17-4e96-8940-e8bb1f7e52e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:08.360448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df401a33-6a17-4e96-8940-e8bb1f7e52e9 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:08.360986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f64a15-acca-4056-9973-a681c5e5b2fc None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:08.361414 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df401a33-6a17-4e96-8940-e8bb1f7e52e9 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:08.361935 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df401a33-6a17-4e96-8940-e8bb1f7e52e9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:08.363891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df401a33-6a17-4e96-8940-e8bb1f7e52e9 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:08.391050 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 373/742] 10.222.0.27 () {56 vars in 1135 bytes} [Wed Sep 7 03:48:08 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:08.397049 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eac778-d063-4a16-ad6b-89e0db48c53d None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:08.397615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eac778-d063-4a16-ad6b-89e0db48c53d None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:08.398070 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eac778-d063-4a16-ad6b-89e0db48c53d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:08.414850 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9a3845-fafd-48c5-94e2-92832d820083 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:08.415492 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9a3845-fafd-48c5-94e2-92832d820083 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:08.415777 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9a3845-fafd-48c5-94e2-92832d820083 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:08.416052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9a3845-fafd-48c5-94e2-92832d820083 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:08.419027 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-39eac778-d063-4a16-ad6b-89e0db48c53d None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:08.421407 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-39eac778-d063-4a16-ad6b-89e0db48c53d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:08.430809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9a3845-fafd-48c5-94e2-92832d820083 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:08.432966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9a3845-fafd-48c5-94e2-92832d820083 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:08.448524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-ac9a3845-fafd-48c5-94e2-92832d820083 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:08.448524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:08.448524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:08.448524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:08.448524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:08.448524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:08.448524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:08.448524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:08.448524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:08.448524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:08.448524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:08.448524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:08.448524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:08.448524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:08.448524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:08.448524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:08.449510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:08.450837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:08.450837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:08.450837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:08.450837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 370/743] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:08.450837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9a3845-fafd-48c5-94e2-92832d820083 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:08.511726 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 374/744] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:08 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:08.512799 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39eac778-d063-4a16-ad6b-89e0db48c53d None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:08.607915 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:08.611712 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9a3845-fafd-48c5-94e2-92832d820083 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:08.630739 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:08.656234 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8e5dd5-5131-434e-8f70-5a80f1c51682 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:08.657140 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8e5dd5-5131-434e-8f70-5a80f1c51682 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:08.657590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8e5dd5-5131-434e-8f70-5a80f1c51682 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:08.658004 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8e5dd5-5131-434e-8f70-5a80f1c51682 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:08.704862 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39eac778-d063-4a16-ad6b-89e0db48c53d None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:08.767422 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37730bda-917d-4533-bb4e-6e80d5dc735e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:08.768378 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37730bda-917d-4533-bb4e-6e80d5dc735e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:08.768879 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37730bda-917d-4533-bb4e-6e80d5dc735e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:08.769350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37730bda-917d-4533-bb4e-6e80d5dc735e None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:08.770251 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37730bda-917d-4533-bb4e-6e80d5dc735e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:08.770837 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37730bda-917d-4533-bb4e-6e80d5dc735e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:08.772424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37730bda-917d-4533-bb4e-6e80d5dc735e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:08.775145 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8e5dd5-5131-434e-8f70-5a80f1c51682 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:08.777307 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8e5dd5-5131-434e-8f70-5a80f1c51682 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:08.786844 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 375/745] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:08.799243 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37730bda-917d-4533-bb4e-6e80d5dc735e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:08.890973 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:08.895907 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 371/746] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:48:08 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:48:08.901705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8e5dd5-5131-434e-8f70-5a80f1c51682 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:08.952102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37730bda-917d-4533-bb4e-6e80d5dc735e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:08.968849 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:08.994674 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574876c6-b9a4-4677-aa78-b69804030a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:08.995546 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574876c6-b9a4-4677-aa78-b69804030a57 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:08.996048 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574876c6-b9a4-4677-aa78-b69804030a57 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:08.996513 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574876c6-b9a4-4677-aa78-b69804030a57 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:09.004478 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574876c6-b9a4-4677-aa78-b69804030a57 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:09.006227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574876c6-b9a4-4677-aa78-b69804030a57 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:09.018338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-574876c6-b9a4-4677-aa78-b69804030a57 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:09.018338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:09.018338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:09.018338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:09.018338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:09.018338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:09.018338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:09.018338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:09.018338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:09.018338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:09.018338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:09.018338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:09.018338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:09.018338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:09.018338 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:09.019219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:09.020190 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:09.020190 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:09.020190 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:09.020190 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:09.020190 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:09.020190 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:09.020190 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:09.020190 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:09.022617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 376/747] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:09.023774 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574876c6-b9a4-4677-aa78-b69804030a57 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:09.031613 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8e5dd5-5131-434e-8f70-5a80f1c51682 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:09.079375 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fd3225-cd03-4124-b563-2d99490c21a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:09.080013 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fd3225-cd03-4124-b563-2d99490c21a9 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:09.080293 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fd3225-cd03-4124-b563-2d99490c21a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:09.080568 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fd3225-cd03-4124-b563-2d99490c21a9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:09.094668 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fd3225-cd03-4124-b563-2d99490c21a9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:09.096813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fd3225-cd03-4124-b563-2d99490c21a9 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:09.113360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-f7fd3225-cd03-4124-b563-2d99490c21a9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:09.113360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:09.113360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:09.113360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:09.113360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:09.113360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:09.113360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:09.113360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:09.113360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:09.113360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:09.113360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:09.113360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:09.113360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:09.113360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:09.113360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:09.113360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:09.114628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:09.115525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:09.115525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:09.115525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:09.121669 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 372/748] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:09.123001 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fd3225-cd03-4124-b563-2d99490c21a9 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:09.140253 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574876c6-b9a4-4677-aa78-b69804030a57 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:09.183816 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:09.187231 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4ef113-8082-423e-a2e7-653a518e0d55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:09.188113 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4ef113-8082-423e-a2e7-653a518e0d55 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:09.188613 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4ef113-8082-423e-a2e7-653a518e0d55 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:09.189099 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4ef113-8082-423e-a2e7-653a518e0d55 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:09.202981 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4ef113-8082-423e-a2e7-653a518e0d55 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:09.209562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-1f4ef113-8082-423e-a2e7-653a518e0d55 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:48:09.209562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:09.209562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:09.209562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:09.209562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:09.209562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:09.209562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:09.209562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:09.209562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:09.209562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:09.209562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:09.209562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:09.209562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:09.209562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:09.209562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:09.209562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:09.210614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:09.210614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:09.210614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:09.210614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:09.210614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:09.210614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:09.210614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:09.210614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:09.210614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:09.210614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:09.210614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:09.210614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:09.210614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:09.210614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:09.210614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:09.210614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:09.211446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:09.211446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:09.211446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:09.211446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:09.211446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:09.211446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:09.211446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:09.211446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:09.211446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:09.211446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:09.211446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:48:09.211446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:09.219082 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 377/749] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:48:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:09.220300 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4ef113-8082-423e-a2e7-653a518e0d55 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:09.245651 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fd3225-cd03-4124-b563-2d99490c21a9 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:09.271996 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:09.289659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a193d23b-a108-45cb-99a2-3a1e0619a504 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:09.290711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a193d23b-a108-45cb-99a2-3a1e0619a504 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:09.291194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a193d23b-a108-45cb-99a2-3a1e0619a504 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:09.291764 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a193d23b-a108-45cb-99a2-3a1e0619a504 None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:09.292787 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a193d23b-a108-45cb-99a2-3a1e0619a504 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:09.293308 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a193d23b-a108-45cb-99a2-3a1e0619a504 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:09.295513 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a193d23b-a108-45cb-99a2-3a1e0619a504 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:09.309500 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 373/750] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:09.310886 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a193d23b-a108-45cb-99a2-3a1e0619a504 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:09.351327 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4ef113-8082-423e-a2e7-653a518e0d55 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:09.386823 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c20a68c-6e39-4140-aa33-beb9b3aa6a8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:09.387792 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c20a68c-6e39-4140-aa33-beb9b3aa6a8c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:09.388832 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c20a68c-6e39-4140-aa33-beb9b3aa6a8c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:09.389367 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c20a68c-6e39-4140-aa33-beb9b3aa6a8c None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:09.390283 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c20a68c-6e39-4140-aa33-beb9b3aa6a8c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:09.390802 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c20a68c-6e39-4140-aa33-beb9b3aa6a8c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:09.392361 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c20a68c-6e39-4140-aa33-beb9b3aa6a8c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:09.409976 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 378/751] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:09.411132 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c20a68c-6e39-4140-aa33-beb9b3aa6a8c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:09.437607 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a193d23b-a108-45cb-99a2-3a1e0619a504 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:09.452484 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:09.484181 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973a4f74-b255-4dd4-9d74-ca9f720be074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:09.484805 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973a4f74-b255-4dd4-9d74-ca9f720be074 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:09.485090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973a4f74-b255-4dd4-9d74-ca9f720be074 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:09.485370 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973a4f74-b255-4dd4-9d74-ca9f720be074 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:09.486485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973a4f74-b255-4dd4-9d74-ca9f720be074 None admin] RBAC: Adding query filter params (name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:09.486794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973a4f74-b255-4dd4-9d74-ca9f720be074 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:09.488705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973a4f74-b255-4dd4-9d74-ca9f720be074 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:09.505782 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 374/752] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:48:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:09.505782 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973a4f74-b255-4dd4-9d74-ca9f720be074 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:09.528673 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:09.535952 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c20a68c-6e39-4140-aa33-beb9b3aa6a8c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:09.573881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0bb6a75-163d-4393-83d4-deea6ef619cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:09.574682 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bb6a75-163d-4393-83d4-deea6ef619cb None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:09.575037 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bb6a75-163d-4393-83d4-deea6ef619cb None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:09.576173 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bb6a75-163d-4393-83d4-deea6ef619cb None admin] PATH_INFO: `/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/953cb6c24bcc4551b91ce540d18cd751/roles/9d5b19a61c46475d8048b1bd66ca25a1` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:09.577972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0bb6a75-163d-4393-83d4-deea6ef619cb None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4816e95144485e8135e5b8f68c3662, user_id=953cb6c24bcc4551b91ce540d18cd751, role_id=9d5b19a61c46475d8048b1bd66ca25a1)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:09.579759 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0bb6a75-163d-4393-83d4-deea6ef619cb None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:09.596347 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 379/753] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:48:09 2022] PUT /identity/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/953cb6c24bcc4551b91ce540d18cd751/roles/9d5b19a61c46475d8048b1bd66ca25a1 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:48:09.597570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0bb6a75-163d-4393-83d4-deea6ef619cb None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:09.636480 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973a4f74-b255-4dd4-9d74-ca9f720be074 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:09.677624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de0ff4f-956b-41ff-8f51-80aa786299ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:09.678574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de0ff4f-956b-41ff-8f51-80aa786299ac None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:09.679003 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de0ff4f-956b-41ff-8f51-80aa786299ac None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:09.679502 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de0ff4f-956b-41ff-8f51-80aa786299ac None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:09.688583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de0ff4f-956b-41ff-8f51-80aa786299ac None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:09.690420 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de0ff4f-956b-41ff-8f51-80aa786299ac None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:09.705024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-0de0ff4f-956b-41ff-8f51-80aa786299ac None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:09.705024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:09.705024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:09.705024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:09.705024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:09.705024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:09.705024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:09.705024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:09.705024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:09.705024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:09.705024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:09.705024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:09.705024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:09.705024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:09.705024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:09.706018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:09.706950 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:09.706950 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:09.706950 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:09.706950 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:09.706950 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:09.706950 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:09.706950 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:09.706950 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:09.708853 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 375/754] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:09.719311 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de0ff4f-956b-41ff-8f51-80aa786299ac None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:09.742160 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0bb6a75-163d-4393-83d4-deea6ef619cb None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:09.810643 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:09.819456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d74483f-fc87-478b-ad1d-ea1d52d3771f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:09.820122 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d74483f-fc87-478b-ad1d-ea1d52d3771f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:09.820400 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d74483f-fc87-478b-ad1d-ea1d52d3771f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:09.820671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d74483f-fc87-478b-ad1d-ea1d52d3771f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:09.835518 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d74483f-fc87-478b-ad1d-ea1d52d3771f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:09.837134 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d74483f-fc87-478b-ad1d-ea1d52d3771f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:09.856291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-1d74483f-fc87-478b-ad1d-ea1d52d3771f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:09.856291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:09.856291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:09.856291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:09.856291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:09.856291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:09.856291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:09.856291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:09.856291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:09.856291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:09.856291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:09.856291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:09.856291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:09.856291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:09.856291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:09.856291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:09.857387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:09.858308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:09.858308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:09.858308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:09.858308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 380/755] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:09.866530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d74483f-fc87-478b-ad1d-ea1d52d3771f None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:09.886745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de0ff4f-956b-41ff-8f51-80aa786299ac None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:09.887400 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:09.944888 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd844459-87ea-480e-ab9c-25921c7df19c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:09.945495 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd844459-87ea-480e-ab9c-25921c7df19c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:09.945771 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd844459-87ea-480e-ab9c-25921c7df19c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:09.946039 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd844459-87ea-480e-ab9c-25921c7df19c None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:09.947594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd844459-87ea-480e-ab9c-25921c7df19c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:09.947897 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd844459-87ea-480e-ab9c-25921c7df19c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:09.949325 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd844459-87ea-480e-ab9c-25921c7df19c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:09.963203 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 376/756] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:09.969877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd844459-87ea-480e-ab9c-25921c7df19c None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:10.056989 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d74483f-fc87-478b-ad1d-ea1d52d3771f None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:10.092970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7881e8fe-2886-4eae-90b9-806451ca7624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:10.093597 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7881e8fe-2886-4eae-90b9-806451ca7624 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:10.093881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7881e8fe-2886-4eae-90b9-806451ca7624 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:10.094184 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7881e8fe-2886-4eae-90b9-806451ca7624 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:10.094915 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7881e8fe-2886-4eae-90b9-806451ca7624 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:10.095228 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7881e8fe-2886-4eae-90b9-806451ca7624 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:10.096680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7881e8fe-2886-4eae-90b9-806451ca7624 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:10.114067 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 381/757] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:10.115150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:10.115397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7881e8fe-2886-4eae-90b9-806451ca7624 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:10.124146 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd844459-87ea-480e-ab9c-25921c7df19c None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:10.161614 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be95fbb3-215d-4969-8bb9-ebc3fbd0bb31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:10.162675 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be95fbb3-215d-4969-8bb9-ebc3fbd0bb31 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:10.163204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be95fbb3-215d-4969-8bb9-ebc3fbd0bb31 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:10.163675 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be95fbb3-215d-4969-8bb9-ebc3fbd0bb31 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:10.164624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be95fbb3-215d-4969-8bb9-ebc3fbd0bb31 None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=f5c3ab9728f74cd8bb2614bcca9e8619) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:10.165144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be95fbb3-215d-4969-8bb9-ebc3fbd0bb31 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:10.166879 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be95fbb3-215d-4969-8bb9-ebc3fbd0bb31 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:10.174145 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 377/758] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:48:09 2022] GET /identity/v3/role_assignments?user.id=f5c3ab9728f74cd8bb2614bcca9e8619&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 259 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:10.194518 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:10.236268 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7881e8fe-2886-4eae-90b9-806451ca7624 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:10.279470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3338dc6-7b09-4d25-b2d1-b830fa4f0fe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:10.280142 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3338dc6-7b09-4d25-b2d1-b830fa4f0fe8 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:10.280428 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3338dc6-7b09-4d25-b2d1-b830fa4f0fe8 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:10.280706 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3338dc6-7b09-4d25-b2d1-b830fa4f0fe8 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:10.290724 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3338dc6-7b09-4d25-b2d1-b830fa4f0fe8 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:10.292220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3338dc6-7b09-4d25-b2d1-b830fa4f0fe8 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:10.305010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-e3338dc6-7b09-4d25-b2d1-b830fa4f0fe8 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 07 03:48:10.305010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:10.305010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:10.305010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:10.305010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:10.305010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:10.305010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:10.305010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:10.305010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:10.305010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:10.305010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:10.305010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:10.305010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:10.305010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:10.305010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:10.305010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:10.306017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:10.307015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:10.307015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:10.307015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:10.307015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:10.307015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:10.307015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:10.307015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:10.307015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:10.307015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 07 03:48:10.307015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:10.308311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 382/759] 10.222.0.27 () {56 vars in 1117 bytes} [Wed Sep 7 03:48:10 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:10.312695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be95fbb3-215d-4969-8bb9-ebc3fbd0bb31 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:10.429728 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be95fbb3-215d-4969-8bb9-ebc3fbd0bb31 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:10.445982 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67c487a-9b9f-40ad-b5ba-c5a5055cc91b None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:10.446317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67c487a-9b9f-40ad-b5ba-c5a5055cc91b None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:10.447068 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67c487a-9b9f-40ad-b5ba-c5a5055cc91b None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:10.449168 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 383/760] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:10.456335 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b589bb57-4100-4051-b1a0-2587095f4242 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:10.456633 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b589bb57-4100-4051-b1a0-2587095f4242 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:10.456877 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b589bb57-4100-4051-b1a0-2587095f4242 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:10.482029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f3b474-096b-4edf-a441-38ad0754708e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:10.482691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f3b474-096b-4edf-a441-38ad0754708e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:10.482971 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f3b474-096b-4edf-a441-38ad0754708e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:10.483238 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f3b474-096b-4edf-a441-38ad0754708e None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:10.483991 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f3b474-096b-4edf-a441-38ad0754708e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:10.484296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f3b474-096b-4edf-a441-38ad0754708e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:10.486523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f3b474-096b-4edf-a441-38ad0754708e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:10.497880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-b589bb57-4100-4051-b1a0-2587095f4242 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:10.499266 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-b589bb57-4100-4051-b1a0-2587095f4242 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:10.507835 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 378/761] 10.222.0.27 () {56 vars in 1149 bytes} [Wed Sep 7 03:48:10 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:10.514929 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f3b474-096b-4edf-a441-38ad0754708e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:10.546622 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:10.618454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 384/762] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:10 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:10.684288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f3b474-096b-4edf-a441-38ad0754708e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:10.739993 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3afaf24-ff8f-40a7-8269-a5c5b75b4e81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:10.741011 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3afaf24-ff8f-40a7-8269-a5c5b75b4e81 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:10.741493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3afaf24-ff8f-40a7-8269-a5c5b75b4e81 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:10.741990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3afaf24-ff8f-40a7-8269-a5c5b75b4e81 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:10.777691 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e06649-ce2e-470b-a52b-c66039ad4d8c None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:10.778009 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e06649-ce2e-470b-a52b-c66039ad4d8c None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:10.778286 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e06649-ce2e-470b-a52b-c66039ad4d8c None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:10.780119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 385/763] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:48:10.785788 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d06a350-f38e-4042-98d6-242bb7d3f515 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:10.786140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d06a350-f38e-4042-98d6-242bb7d3f515 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:10.786368 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d06a350-f38e-4042-98d6-242bb7d3f515 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:10.827280 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-9d06a350-f38e-4042-98d6-242bb7d3f515 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:10.828487 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-9d06a350-f38e-4042-98d6-242bb7d3f515 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:10.830790 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3afaf24-ff8f-40a7-8269-a5c5b75b4e81 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:10.832463 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3afaf24-ff8f-40a7-8269-a5c5b75b4e81 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:10.874471 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:10.919793 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:10.938562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 386/764] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:10 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:48:10.939484 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d06a350-f38e-4042-98d6-242bb7d3f515 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:11.013717 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 379/765] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:48:10 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 495 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:48:11.019060 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3afaf24-ff8f-40a7-8269-a5c5b75b4e81 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:11.098250 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d06a350-f38e-4042-98d6-242bb7d3f515 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:11.142973 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5dfd66-770a-4a48-8d14-12ca440ae857 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:11.144127 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5dfd66-770a-4a48-8d14-12ca440ae857 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:11.145974 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5dfd66-770a-4a48-8d14-12ca440ae857 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:11.147038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5dfd66-770a-4a48-8d14-12ca440ae857 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:11.147826 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5dfd66-770a-4a48-8d14-12ca440ae857 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:11.149611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5dfd66-770a-4a48-8d14-12ca440ae857 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-0d5dfd66-770a-4a48-8d14-12ca440ae857 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:11.164598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:11.165634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:11.165634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:11.165634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:11.165634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:11.165634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:11.165634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:11.165634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:11.165634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:11.165634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:11.165634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:11.165634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:11.165634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:11.165634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:11.165634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:11.165634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:11.165634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:11.166510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:11.166510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:11.166510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:11.166510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:11.166510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:11.166510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:11.166510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:11.166510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:11.166510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:11.166510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:11.166510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:11.166510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:11.166510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 07 03:48:11.166510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:11.167765 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 387/766] 10.222.0.27 () {56 vars in 1129 bytes} [Wed Sep 7 03:48:10 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:11.175767 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5dfd66-770a-4a48-8d14-12ca440ae857 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:11.232496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3afaf24-ff8f-40a7-8269-a5c5b75b4e81 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:11.245411 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:11.252173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:11.272259 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c03a979-1509-4d9b-8e83-032df5ddd828 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:11.272888 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c03a979-1509-4d9b-8e83-032df5ddd828 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:11.273163 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c03a979-1509-4d9b-8e83-032df5ddd828 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:11.273429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c03a979-1509-4d9b-8e83-032df5ddd828 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:11.284871 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c03a979-1509-4d9b-8e83-032df5ddd828 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:11.286496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c03a979-1509-4d9b-8e83-032df5ddd828 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:11.302429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-3c03a979-1509-4d9b-8e83-032df5ddd828 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:11.302429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:11.302429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:11.302429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:11.302429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:11.302429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:11.302429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:11.302429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:11.302429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:11.302429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:11.302429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:11.302429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:11.302429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:11.302429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:11.302429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:11.302429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:11.303385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:11.304333 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:11.304333 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:11.304333 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:11.304333 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 380/767] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:11.305083 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c03a979-1509-4d9b-8e83-032df5ddd828 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:11.312172 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5dfd66-770a-4a48-8d14-12ca440ae857 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:11.349320 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88022e7-10ee-4260-bba0-99b09fd73997 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:11.350328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88022e7-10ee-4260-bba0-99b09fd73997 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:11.350895 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88022e7-10ee-4260-bba0-99b09fd73997 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:11.351367 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88022e7-10ee-4260-bba0-99b09fd73997 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:11.352208 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88022e7-10ee-4260-bba0-99b09fd73997 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:11.352754 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88022e7-10ee-4260-bba0-99b09fd73997 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:11.354377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88022e7-10ee-4260-bba0-99b09fd73997 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:11.364649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 388/768] 10.222.0.27 () {56 vars in 1124 bytes} [Wed Sep 7 03:48:11 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:11.365889 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88022e7-10ee-4260-bba0-99b09fd73997 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:11.414474 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c03a979-1509-4d9b-8e83-032df5ddd828 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:11.450406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f99f02-549c-488e-89f6-4bc155a5767e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:11.451397 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f99f02-549c-488e-89f6-4bc155a5767e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:11.451647 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f99f02-549c-488e-89f6-4bc155a5767e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:11.451916 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f99f02-549c-488e-89f6-4bc155a5767e None admin] PATH_INFO: `/v3/services/network` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:11.452491 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f99f02-549c-488e-89f6-4bc155a5767e None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:11.454398 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f99f02-549c-488e-89f6-4bc155a5767e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-38f99f02-549c-488e-89f6-4bc155a5767e None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:11.466489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:11.467793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:11.467793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:11.467793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:11.467793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:11.467793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:11.467793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:11.467793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:11.467793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:11.467793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:11.467793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:11.467793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:11.467793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:11.467793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:11.467793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:11.467793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:11.467793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:11.468640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:11.468640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:11.468640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:11.468640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:11.468640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:11.468640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:11.468640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:11.468640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:11.468640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:11.468640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:11.468640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:11.468640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:11.468640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 07 03:48:11.468640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:11.473180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 381/769] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:11 2022] GET /identity/v3/services/network => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:11.473820 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f99f02-549c-488e-89f6-4bc155a5767e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:11.483496 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88022e7-10ee-4260-bba0-99b09fd73997 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:11.498593 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:11.507917 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:11.524579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90cd9ba-e962-4efc-816a-f82af1a7819c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:11.525614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90cd9ba-e962-4efc-816a-f82af1a7819c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:11.526274 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90cd9ba-e962-4efc-816a-f82af1a7819c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:11.526750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90cd9ba-e962-4efc-816a-f82af1a7819c None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:11.527590 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90cd9ba-e962-4efc-816a-f82af1a7819c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:11.528107 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90cd9ba-e962-4efc-816a-f82af1a7819c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:11.530931 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90cd9ba-e962-4efc-816a-f82af1a7819c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:11.544045 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 389/770] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:11.545530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90cd9ba-e962-4efc-816a-f82af1a7819c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:11.583143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f99f02-549c-488e-89f6-4bc155a5767e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:11.620150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d10d8b-5e62-4c89-b212-68ece3199fd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:11.620600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d10d8b-5e62-4c89-b212-68ece3199fd6 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:11.620882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d10d8b-5e62-4c89-b212-68ece3199fd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:11.621149 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d10d8b-5e62-4c89-b212-68ece3199fd6 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:11.621781 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d10d8b-5e62-4c89-b212-68ece3199fd6 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:11.622080 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d10d8b-5e62-4c89-b212-68ece3199fd6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:11.623637 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d10d8b-5e62-4c89-b212-68ece3199fd6 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:11.633637 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 382/771] 10.222.0.27 () {56 vars in 1124 bytes} [Wed Sep 7 03:48:11 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:11.633637 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d10d8b-5e62-4c89-b212-68ece3199fd6 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:11.663530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90cd9ba-e962-4efc-816a-f82af1a7819c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:11.702101 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137cd96b-5f51-4095-b239-9e735d478c45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:11.703006 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137cd96b-5f51-4095-b239-9e735d478c45 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:11.703446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137cd96b-5f51-4095-b239-9e735d478c45 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:11.704209 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137cd96b-5f51-4095-b239-9e735d478c45 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:11.705018 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137cd96b-5f51-4095-b239-9e735d478c45 None admin] RBAC: Adding query filter params (name=network) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:11.705512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137cd96b-5f51-4095-b239-9e735d478c45 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:11.707076 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137cd96b-5f51-4095-b239-9e735d478c45 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:11.713693 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 390/772] 10.222.0.27 () {56 vars in 1112 bytes} [Wed Sep 7 03:48:11 2022] GET /identity/v3/services?name=network => generated 124 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:11.714790 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137cd96b-5f51-4095-b239-9e735d478c45 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:11.740696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:11.746702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d10d8b-5e62-4c89-b212-68ece3199fd6 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:11.762769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:11.787243 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1766aad-0e9f-432d-92cb-4df538533404 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:11.788347 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1766aad-0e9f-432d-92cb-4df538533404 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:11.788866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1766aad-0e9f-432d-92cb-4df538533404 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:11.789333 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1766aad-0e9f-432d-92cb-4df538533404 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:11.798062 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1766aad-0e9f-432d-92cb-4df538533404 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:11.799856 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1766aad-0e9f-432d-92cb-4df538533404 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:11.812577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-f1766aad-0e9f-432d-92cb-4df538533404 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:11.812577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:11.812577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:11.812577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:11.812577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:11.812577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:11.812577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:11.812577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:11.812577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:11.812577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:11.812577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:11.812577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:11.812577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:11.812577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:11.812577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:11.813481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:11.814415 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:11.814415 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:11.814415 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:11.814415 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:11.814415 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:11.814415 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:11.814415 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:11.814415 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:11.816864 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 383/773] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:11.817882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1766aad-0e9f-432d-92cb-4df538533404 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:11.830911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137cd96b-5f51-4095-b239-9e735d478c45 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:11.868157 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71127dc-1a94-48ff-a5af-f02713b15f05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:11.868777 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71127dc-1a94-48ff-a5af-f02713b15f05 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:11.869055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71127dc-1a94-48ff-a5af-f02713b15f05 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:11.869324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71127dc-1a94-48ff-a5af-f02713b15f05 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:11.870149 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71127dc-1a94-48ff-a5af-f02713b15f05 None admin] RBAC: Adding query filter params (interface=public, service_id=87c7bd7e7d584322b0b3cda45f496a6e, region_id=RegionOne) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:11.870474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71127dc-1a94-48ff-a5af-f02713b15f05 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:11.871931 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71127dc-1a94-48ff-a5af-f02713b15f05 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:11.885418 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 391/774] 10.222.0.27 () {56 vars in 1251 bytes} [Wed Sep 7 03:48:11 2022] GET /identity/v3/endpoints?service_id=87c7bd7e7d584322b0b3cda45f496a6e&interface=public®ion_id=RegionOne => generated 194 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:11.886383 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71127dc-1a94-48ff-a5af-f02713b15f05 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:11.929747 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1766aad-0e9f-432d-92cb-4df538533404 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:11.967468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e701b6f9-b26c-41f2-875b-924455e78573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:11.968416 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e701b6f9-b26c-41f2-875b-924455e78573 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:11.968937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e701b6f9-b26c-41f2-875b-924455e78573 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:11.969456 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e701b6f9-b26c-41f2-875b-924455e78573 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:11.970311 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e701b6f9-b26c-41f2-875b-924455e78573 None admin] RBAC: Adding query filter params (type=network) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:11.970865 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e701b6f9-b26c-41f2-875b-924455e78573 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:11.972598 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e701b6f9-b26c-41f2-875b-924455e78573 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:11.983775 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 384/775] 10.222.0.27 () {56 vars in 1112 bytes} [Wed Sep 7 03:48:11 2022] GET /identity/v3/services?type=network => generated 350 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:11.984931 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e701b6f9-b26c-41f2-875b-924455e78573 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:11.994935 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71127dc-1a94-48ff-a5af-f02713b15f05 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:11.998467 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:12.008336 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:12.036094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce610f79-4fc2-4311-ad25-e131bf5a8767 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:12.037008 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce610f79-4fc2-4311-ad25-e131bf5a8767 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:12.037450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce610f79-4fc2-4311-ad25-e131bf5a8767 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:12.037960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce610f79-4fc2-4311-ad25-e131bf5a8767 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:12.039078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce610f79-4fc2-4311-ad25-e131bf5a8767 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:12.039536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce610f79-4fc2-4311-ad25-e131bf5a8767 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:12.041201 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce610f79-4fc2-4311-ad25-e131bf5a8767 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:12.056696 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 392/776] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:12.057993 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce610f79-4fc2-4311-ad25-e131bf5a8767 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:12.095799 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e701b6f9-b26c-41f2-875b-924455e78573 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:12.131058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cad266a-5cc3-4b3f-83ad-c8e2ce934e4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:12.131773 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cad266a-5cc3-4b3f-83ad-c8e2ce934e4e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:12.132135 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cad266a-5cc3-4b3f-83ad-c8e2ce934e4e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:12.132479 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cad266a-5cc3-4b3f-83ad-c8e2ce934e4e None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:12.133151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cad266a-5cc3-4b3f-83ad-c8e2ce934e4e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:12.134661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cad266a-5cc3-4b3f-83ad-c8e2ce934e4e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:12.145622 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 385/777] 10.222.0.27 () {56 vars in 1087 bytes} [Wed Sep 7 03:48:11 2022] GET /identity/v3/services => generated 772 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:12.145622 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cad266a-5cc3-4b3f-83ad-c8e2ce934e4e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:12.196308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce610f79-4fc2-4311-ad25-e131bf5a8767 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:12.248375 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:12.251120 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a9bd00-ffe2-498b-a840-c7127d311a5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:12.251759 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a9bd00-ffe2-498b-a840-c7127d311a5f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:12.252071 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a9bd00-ffe2-498b-a840-c7127d311a5f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:12.252350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a9bd00-ffe2-498b-a840-c7127d311a5f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:12.253075 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a9bd00-ffe2-498b-a840-c7127d311a5f None admin] RBAC: Adding query filter params (interface=public, service_id=651d5e49ee1a45abadda6a907ce533c3, region_id=RegionOne) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:12.254558 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a9bd00-ffe2-498b-a840-c7127d311a5f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:12.256659 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a9bd00-ffe2-498b-a840-c7127d311a5f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:12.265377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 393/778] 10.222.0.27 () {56 vars in 1251 bytes} [Wed Sep 7 03:48:12 2022] GET /identity/v3/endpoints?service_id=651d5e49ee1a45abadda6a907ce533c3&interface=public®ion_id=RegionOne => generated 194 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:12.270176 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52983f2d-e2d8-4942-9f10-d26b895bf99d None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:12.270176 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52983f2d-e2d8-4942-9f10-d26b895bf99d None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:12.270176 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52983f2d-e2d8-4942-9f10-d26b895bf99d None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:12.273589 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 394/779] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:12 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:12.274853 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52983f2d-e2d8-4942-9f10-d26b895bf99d None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:12.276602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cad266a-5cc3-4b3f-83ad-c8e2ce934e4e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:12.306887 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:12.325869 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cf81ca-453d-43da-b5e9-7116e2637bc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:12.325869 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cf81ca-453d-43da-b5e9-7116e2637bc3 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:12.325869 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cf81ca-453d-43da-b5e9-7116e2637bc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:12.325869 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cf81ca-453d-43da-b5e9-7116e2637bc3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:12.326361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cf81ca-453d-43da-b5e9-7116e2637bc3 None admin] RBAC: Adding query filter params (role.id=a7b16bb097f14f78a7a31c04bd6d32bd, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=e7b8f511ee884dbfb4f56af78bf75984) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:12.326361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cf81ca-453d-43da-b5e9-7116e2637bc3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:12.326361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cf81ca-453d-43da-b5e9-7116e2637bc3 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:12.331213 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 386/780] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:48:12 2022] GET /identity/v3/role_assignments?user.id=e7b8f511ee884dbfb4f56af78bf75984&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=a7b16bb097f14f78a7a31c04bd6d32bd => generated 259 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:12.332580 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbe81af-3836-4867-963f-059f3528a711 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:12.332875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbe81af-3836-4867-963f-059f3528a711 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:12.333157 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbe81af-3836-4867-963f-059f3528a711 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:12.354498 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-abbe81af-3836-4867-963f-059f3528a711 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:12.355710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-abbe81af-3836-4867-963f-059f3528a711 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:12.415526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52983f2d-e2d8-4942-9f10-d26b895bf99d None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:12.442573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 387/781] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:12 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:12.457477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7cfb9b6-7ca0-44e2-8101-a20f5df39905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:12.458519 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cfb9b6-7ca0-44e2-8101-a20f5df39905 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:12.458997 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cfb9b6-7ca0-44e2-8101-a20f5df39905 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:12.459464 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cfb9b6-7ca0-44e2-8101-a20f5df39905 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:12.460325 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7cfb9b6-7ca0-44e2-8101-a20f5df39905 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:12.461943 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7cfb9b6-7ca0-44e2-8101-a20f5df39905 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:12.468840 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 395/782] 10.222.0.27 () {56 vars in 1087 bytes} [Wed Sep 7 03:48:12 2022] GET /identity/v3/services => generated 772 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:12.717902 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbe81af-3836-4867-963f-059f3528a711 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:12.800204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:12.804910 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f913f7-a8a7-408b-b97e-b0046663ecf1 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:12.805446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f913f7-a8a7-408b-b97e-b0046663ecf1 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:12.805722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f913f7-a8a7-408b-b97e-b0046663ecf1 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:12.808165 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 396/783] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:12.817448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e87fdf-42a5-4d80-9fe7-1f79af548779 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:12.817731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e87fdf-42a5-4d80-9fe7-1f79af548779 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:12.818002 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e87fdf-42a5-4d80-9fe7-1f79af548779 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:12.826259 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbe81af-3836-4867-963f-059f3528a711 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:12.839417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-84e87fdf-42a5-4d80-9fe7-1f79af548779 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:12.840731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-84e87fdf-42a5-4d80-9fe7-1f79af548779 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:12.870459 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172f18b7-3233-4afa-897d-994a3f41c2e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:12.871359 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172f18b7-3233-4afa-897d-994a3f41c2e5 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:12.871638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172f18b7-3233-4afa-897d-994a3f41c2e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:12.871905 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172f18b7-3233-4afa-897d-994a3f41c2e5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:12.884849 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172f18b7-3233-4afa-897d-994a3f41c2e5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:12.890440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-172f18b7-3233-4afa-897d-994a3f41c2e5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:12.890440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:12.890440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:12.890440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:12.890440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:12.890440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:12.890440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:12.890440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:12.890440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:12.890440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:12.890440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:12.890440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:12.890440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:12.890440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:12.890440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:12.890440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:12.891383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:12.891383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:12.891383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:12.891383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:12.891383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:12.891383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:12.891383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:12.891383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:12.891383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:12.891383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:12.891383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:12.891383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:12.891383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:12.891383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:12.891383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:12.891383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:12.892200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:12.892200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:12.892200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:12.892200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:12.892200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:12.892200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:12.892200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:12.892200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:12.892200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:12.892200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:12.892200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:12.892200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:12.892200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 388/784] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:48:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:12.899153 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172f18b7-3233-4afa-897d-994a3f41c2e5 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:12.942708 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 397/785] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:12 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:13.045617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172f18b7-3233-4afa-897d-994a3f41c2e5 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:13.092136 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bf89af-309c-4e0a-ba60-d8347ba08ab2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:13.092841 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bf89af-309c-4e0a-ba60-d8347ba08ab2 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:13.093125 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bf89af-309c-4e0a-ba60-d8347ba08ab2 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:13.093397 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bf89af-309c-4e0a-ba60-d8347ba08ab2 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:13.094062 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bf89af-309c-4e0a-ba60-d8347ba08ab2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:13.094463 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bf89af-309c-4e0a-ba60-d8347ba08ab2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:13.095824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bf89af-309c-4e0a-ba60-d8347ba08ab2 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:13.104169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:13.109142 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 389/786] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:48:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:13.114983 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e87fdf-42a5-4d80-9fe7-1f79af548779 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:13.140618 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:13.232038 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bf89af-309c-4e0a-ba60-d8347ba08ab2 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:13.399354 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e87fdf-42a5-4d80-9fe7-1f79af548779 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:13.420203 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bf89af-309c-4e0a-ba60-d8347ba08ab2 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:13.463298 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fd0bec-e631-494a-b149-9fea19e8cbaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:13.464350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fd0bec-e631-494a-b149-9fea19e8cbaa None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:13.464850 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fd0bec-e631-494a-b149-9fea19e8cbaa None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:13.467671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fd0bec-e631-494a-b149-9fea19e8cbaa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:13.485238 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fd0bec-e631-494a-b149-9fea19e8cbaa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:13.487532 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fd0bec-e631-494a-b149-9fea19e8cbaa None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:13.490058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6aa6de-e37b-4752-8559-93661c5d3cea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:13.491286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6aa6de-e37b-4752-8559-93661c5d3cea None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:13.491758 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6aa6de-e37b-4752-8559-93661c5d3cea None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:13.492294 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6aa6de-e37b-4752-8559-93661c5d3cea None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:13.511512 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6aa6de-e37b-4752-8559-93661c5d3cea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:13.517875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-bf6aa6de-e37b-4752-8559-93661c5d3cea None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:13.517875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:13.517875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:13.517875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:13.517875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:13.517875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:13.517875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:13.517875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:13.517875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:13.517875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:13.517875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:13.517875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:13.517875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:13.517875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:13.517875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:13.517875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:13.518828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:13.518828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:13.518828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:13.518828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:13.518828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:13.518828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:13.518828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:13.518828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:13.518828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:13.518828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:13.518828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:13.518828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:13.518828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:13.518828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:13.518828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:13.518828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:13.519673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:13.519673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:13.519673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:13.519673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:13.519673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:13.519673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:13.519673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:13.519673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:13.519673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:13.519673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:13.519673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:13.519673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:13.520324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-c4fd0bec-e631-494a-b149-9fea19e8cbaa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:13.520324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:13.520324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:13.520324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:13.520324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:13.520324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:13.520324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:13.520324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:13.520324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:13.520324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:13.520324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:13.520324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:13.520324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:13.520324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:13.520324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:13.520324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:13.521153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:13.522046 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:13.522046 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:13.522046 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:13.529457 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 390/788] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:48:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 298 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:13.529651 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 398/788] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 407 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:13.535632 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fd0bec-e631-494a-b149-9fea19e8cbaa None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:13.535937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6aa6de-e37b-4752-8559-93661c5d3cea None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:13.623014 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:13.674339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6aa6de-e37b-4752-8559-93661c5d3cea None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:13.710662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f4c97a-cc39-4162-8308-db1b7ed19983 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:13.711904 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f4c97a-cc39-4162-8308-db1b7ed19983 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:13.712386 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f4c97a-cc39-4162-8308-db1b7ed19983 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:13.712862 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f4c97a-cc39-4162-8308-db1b7ed19983 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:13.714996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f4c97a-cc39-4162-8308-db1b7ed19983 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:13.715530 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f4c97a-cc39-4162-8308-db1b7ed19983 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:13.718104 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f4c97a-cc39-4162-8308-db1b7ed19983 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:13.727639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 391/789] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:48:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:13.734291 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f4c97a-cc39-4162-8308-db1b7ed19983 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:13.760328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:13.822022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fd0bec-e631-494a-b149-9fea19e8cbaa None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:13.898666 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce39792-f688-48b9-8313-52fd5b98cd11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:13.899679 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce39792-f688-48b9-8313-52fd5b98cd11 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:13.900167 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce39792-f688-48b9-8313-52fd5b98cd11 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:13.901402 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce39792-f688-48b9-8313-52fd5b98cd11 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:13.902290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce39792-f688-48b9-8313-52fd5b98cd11 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:13.903527 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce39792-f688-48b9-8313-52fd5b98cd11 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:13.906712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce39792-f688-48b9-8313-52fd5b98cd11 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:13.924944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 399/790] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 389 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:13.930627 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce39792-f688-48b9-8313-52fd5b98cd11 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:13.956553 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f4c97a-cc39-4162-8308-db1b7ed19983 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:14.006452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:14.030854 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395dbb84-5acf-4542-af5d-7bdc5b6ca3ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:14.031949 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395dbb84-5acf-4542-af5d-7bdc5b6ca3ca None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:14.032471 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395dbb84-5acf-4542-af5d-7bdc5b6ca3ca None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:14.033079 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395dbb84-5acf-4542-af5d-7bdc5b6ca3ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:14.047866 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce39792-f688-48b9-8313-52fd5b98cd11 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:14.056118 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395dbb84-5acf-4542-af5d-7bdc5b6ca3ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:14.057938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395dbb84-5acf-4542-af5d-7bdc5b6ca3ca None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:14.082032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:14.085389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-395dbb84-5acf-4542-af5d-7bdc5b6ca3ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:14.085389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:14.085389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:14.085389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:14.085389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:14.085389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:14.085389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:14.085389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:14.085389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:14.085389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:14.085389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:14.085389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:14.085389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:14.085389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:14.085389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:14.085389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:14.086706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:14.087792 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:14.087792 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:14.087792 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:14.090971 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 392/791] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 357 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:14.096705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1fa91e-7a01-4f85-ac96-bdd8f755441b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:14.097600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1fa91e-7a01-4f85-ac96-bdd8f755441b None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:14.098077 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1fa91e-7a01-4f85-ac96-bdd8f755441b None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:14.098702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1fa91e-7a01-4f85-ac96-bdd8f755441b None admin] PATH_INFO: `/v3/users/nova` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:14.104285 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395dbb84-5acf-4542-af5d-7bdc5b6ca3ca None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:14.119526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1fa91e-7a01-4f85-ac96-bdd8f755441b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:14.124351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1fa91e-7a01-4f85-ac96-bdd8f755441b None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:14.152561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-fc1fa91e-7a01-4f85-ac96-bdd8f755441b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:48:14.152561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:14.152561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:14.152561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:14.152561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:14.152561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:14.152561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:14.152561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:14.152561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:14.152561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:14.152561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:14.152561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:14.152561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:14.152561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:14.152561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:14.152561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:14.154090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:14.155511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:14.155511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:14.155511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:14.155511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:14.155511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:14.155511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:14.155511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:14.155511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:14.155511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:48:14.155511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:14.163634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 400/792] 10.222.0.27 () {56 vars in 1093 bytes} [Wed Sep 7 03:48:13 2022] GET /identity/v3/users/nova => generated 82 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:14.180011 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1fa91e-7a01-4f85-ac96-bdd8f755441b None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:14.315059 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395dbb84-5acf-4542-af5d-7bdc5b6ca3ca None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:14.387440 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1fa91e-7a01-4f85-ac96-bdd8f755441b None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:14.391022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28042523-5f76-4b3a-a9a6-b8e93a7ede8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:14.391636 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28042523-5f76-4b3a-a9a6-b8e93a7ede8d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:14.391907 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28042523-5f76-4b3a-a9a6-b8e93a7ede8d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:14.392173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28042523-5f76-4b3a-a9a6-b8e93a7ede8d None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:14.392816 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28042523-5f76-4b3a-a9a6-b8e93a7ede8d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:14.393116 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28042523-5f76-4b3a-a9a6-b8e93a7ede8d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:14.394501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28042523-5f76-4b3a-a9a6-b8e93a7ede8d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:14.424938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 393/793] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:14.444592 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28042523-5f76-4b3a-a9a6-b8e93a7ede8d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:14.462412 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e553178-f3f3-4693-bc1b-94c5c8641211 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:14.463052 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e553178-f3f3-4693-bc1b-94c5c8641211 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:14.463300 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e553178-f3f3-4693-bc1b-94c5c8641211 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:14.465399 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e553178-f3f3-4693-bc1b-94c5c8641211 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:14.466827 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e553178-f3f3-4693-bc1b-94c5c8641211 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:14.467345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e553178-f3f3-4693-bc1b-94c5c8641211 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:14.468795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e553178-f3f3-4693-bc1b-94c5c8641211 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:14.506472 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:14.515943 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 401/794] 10.222.0.27 () {56 vars in 1133 bytes} [Wed Sep 7 03:48:14 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:14.521258 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e553178-f3f3-4693-bc1b-94c5c8641211 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:14.598550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:14.645797 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e553178-f3f3-4693-bc1b-94c5c8641211 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:14.686344 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4908c1f-d230-43be-b670-96e067cf9fbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:14.687295 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4908c1f-d230-43be-b670-96e067cf9fbc None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:14.687593 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4908c1f-d230-43be-b670-96e067cf9fbc None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:14.687865 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4908c1f-d230-43be-b670-96e067cf9fbc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:14.716248 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28042523-5f76-4b3a-a9a6-b8e93a7ede8d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:14.793305 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dcf3b6-3c58-4cac-a692-4dbdd963f66f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:14.794569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dcf3b6-3c58-4cac-a692-4dbdd963f66f None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:14.795087 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dcf3b6-3c58-4cac-a692-4dbdd963f66f None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:14.795399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dcf3b6-3c58-4cac-a692-4dbdd963f66f None admin] PATH_INFO: `/v3/users/swift` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:14.796823 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4908c1f-d230-43be-b670-96e067cf9fbc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:14.798443 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4908c1f-d230-43be-b670-96e067cf9fbc None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:14.804910 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dcf3b6-3c58-4cac-a692-4dbdd963f66f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:14.807022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dcf3b6-3c58-4cac-a692-4dbdd963f66f None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:14.822837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-a4dcf3b6-3c58-4cac-a692-4dbdd963f66f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 07 03:48:14.822837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:14.822837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:14.822837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:14.822837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:14.822837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:14.822837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:14.822837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:14.822837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:14.822837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:14.822837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:14.822837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:14.822837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:14.822837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:14.822837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:14.822837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:14.823809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:14.824740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:14.824740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:14.824740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:14.824740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:14.824740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:14.824740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:14.824740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:14.824740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:14.824740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 07 03:48:14.824740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:14.834304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 394/795] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:48:14 2022] GET /identity/v3/users/swift => generated 83 bytes in 383 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:14.834304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b673cf71-32e2-4391-882d-80cf579b7752 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:14.834304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b673cf71-32e2-4391-882d-80cf579b7752 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:14.834304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b673cf71-32e2-4391-882d-80cf579b7752 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:14.834304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 395/796] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:14.838089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b673cf71-32e2-4391-882d-80cf579b7752 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:14.870543 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:14.915982 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 402/797] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:48:14 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:48:14.917334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2136a4ef-dd12-4148-9edf-d8a119484206 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:14.917685 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2136a4ef-dd12-4148-9edf-d8a119484206 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:14.918025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2136a4ef-dd12-4148-9edf-d8a119484206 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:14.939804 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-2136a4ef-dd12-4148-9edf-d8a119484206 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:14.941436 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-2136a4ef-dd12-4148-9edf-d8a119484206 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:14.949420 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:14.958121 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b673cf71-32e2-4391-882d-80cf579b7752 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:14.993180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7c6838-8c75-4d1a-b38d-980c62e3af8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:14.993706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7c6838-8c75-4d1a-b38d-980c62e3af8c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:14.993984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7c6838-8c75-4d1a-b38d-980c62e3af8c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:14.994286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7c6838-8c75-4d1a-b38d-980c62e3af8c None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:14.995061 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7c6838-8c75-4d1a-b38d-980c62e3af8c None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:14.995319 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7c6838-8c75-4d1a-b38d-980c62e3af8c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:14.996725 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7c6838-8c75-4d1a-b38d-980c62e3af8c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:15.019444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 403/798] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:14 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:15.021073 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffe8133-2c2c-4b05-a8d6-bc7441b34e3b None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:15.021354 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffe8133-2c2c-4b05-a8d6-bc7441b34e3b None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:15.021626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffe8133-2c2c-4b05-a8d6-bc7441b34e3b None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:15.022278 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 396/799] 10.222.0.27 () {56 vars in 1135 bytes} [Wed Sep 7 03:48:14 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:15.023302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 404/800] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:15.023623 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7c6838-8c75-4d1a-b38d-980c62e3af8c None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:15.024003 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffe8133-2c2c-4b05-a8d6-bc7441b34e3b None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:15.136957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:15.146431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7c6838-8c75-4d1a-b38d-980c62e3af8c None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:15.165864 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffe8133-2c2c-4b05-a8d6-bc7441b34e3b None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:15.183674 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814849c7-fb42-4094-a840-824092fd67d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:15.184414 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814849c7-fb42-4094-a840-824092fd67d7 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:15.184781 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814849c7-fb42-4094-a840-824092fd67d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:15.185184 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814849c7-fb42-4094-a840-824092fd67d7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:15.195984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814849c7-fb42-4094-a840-824092fd67d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:15.197500 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814849c7-fb42-4094-a840-824092fd67d7 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:15.210483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437d4fcd-b049-442c-9f2e-23f437fd5792 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:15.211385 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437d4fcd-b049-442c-9f2e-23f437fd5792 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:15.211650 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437d4fcd-b049-442c-9f2e-23f437fd5792 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:15.211929 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437d4fcd-b049-442c-9f2e-23f437fd5792 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:15.220154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-814849c7-fb42-4094-a840-824092fd67d7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:15.220154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:15.220154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:15.220154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:15.220154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:15.220154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:15.220154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:15.220154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:15.220154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:15.220154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:15.220154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:15.220154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:15.220154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:15.220154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:15.220154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:15.220154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:15.221092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:15.221971 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:15.221971 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:15.221971 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:15.222783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 397/801] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:15.224059 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffc6a34-6fdf-4c94-8079-3ceb0c00e82f None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:15.224345 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffc6a34-6fdf-4c94-8079-3ceb0c00e82f None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:15.224620 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffc6a34-6fdf-4c94-8079-3ceb0c00e82f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:15.227163 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437d4fcd-b049-442c-9f2e-23f437fd5792 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:15.228802 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437d4fcd-b049-442c-9f2e-23f437fd5792 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:15.231965 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:15.243373 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-5ffc6a34-6fdf-4c94-8079-3ceb0c00e82f None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:15.244374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-437d4fcd-b049-442c-9f2e-23f437fd5792 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:15.244374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:15.244374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:15.244374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:15.244374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:15.244374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:15.244374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:15.244374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:15.244374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:15.244374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:15.244374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:15.244374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:15.244374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:15.244374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:15.244374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:15.244374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:15.245289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:15.246594 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:15.246594 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:15.246594 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:15.246773 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-5ffc6a34-6fdf-4c94-8079-3ceb0c00e82f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:15.250185 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 405/802] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:15.251485 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34c31bf-f4f9-4311-a2c0-919c1d5b94a6 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:15.251769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34c31bf-f4f9-4311-a2c0-919c1d5b94a6 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:15.252041 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34c31bf-f4f9-4311-a2c0-919c1d5b94a6 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:15.253428 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 406/803] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:15.253969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34c31bf-f4f9-4311-a2c0-919c1d5b94a6 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:15.323519 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 398/804] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:15 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:15.324370 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffc6a34-6fdf-4c94-8079-3ceb0c00e82f None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:15.375584 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34c31bf-f4f9-4311-a2c0-919c1d5b94a6 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:15.406990 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:15.413829 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30116dc-0e8f-44ba-b7dd-b59b556a8bed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:15.414500 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30116dc-0e8f-44ba-b7dd-b59b556a8bed None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:15.414928 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30116dc-0e8f-44ba-b7dd-b59b556a8bed None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:15.415207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30116dc-0e8f-44ba-b7dd-b59b556a8bed None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:15.415880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30116dc-0e8f-44ba-b7dd-b59b556a8bed None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:15.416190 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30116dc-0e8f-44ba-b7dd-b59b556a8bed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:15.417576 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30116dc-0e8f-44ba-b7dd-b59b556a8bed None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:15.426287 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffc6a34-6fdf-4c94-8079-3ceb0c00e82f None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:15.430161 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 407/805] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:15.431304 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7443a774-0bcc-4612-9e2c-a023f897f40c None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:15.431591 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7443a774-0bcc-4612-9e2c-a023f897f40c None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:15.431867 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7443a774-0bcc-4612-9e2c-a023f897f40c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:15.451222 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-7443a774-0bcc-4612-9e2c-a023f897f40c None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:15.452407 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-7443a774-0bcc-4612-9e2c-a023f897f40c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:15.469049 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e92bc3-8167-4e2d-a38d-aa103f7dd271 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:15.469829 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e92bc3-8167-4e2d-a38d-aa103f7dd271 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:15.470410 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e92bc3-8167-4e2d-a38d-aa103f7dd271 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:15.470519 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e92bc3-8167-4e2d-a38d-aa103f7dd271 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:15.471214 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e92bc3-8167-4e2d-a38d-aa103f7dd271 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:15.471521 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e92bc3-8167-4e2d-a38d-aa103f7dd271 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:15.473101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e92bc3-8167-4e2d-a38d-aa103f7dd271 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:15.474688 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:15.487315 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 399/806] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:15.488203 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e92bc3-8167-4e2d-a38d-aa103f7dd271 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:15.525508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 408/807] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:15 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:15.526977 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7443a774-0bcc-4612-9e2c-a023f897f40c None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:15.645400 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7443a774-0bcc-4612-9e2c-a023f897f40c None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:15.671875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:15.686655 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6c369b-af50-433b-a73d-3b6b9c8dd7be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:15.687371 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6c369b-af50-433b-a73d-3b6b9c8dd7be None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:15.687576 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6c369b-af50-433b-a73d-3b6b9c8dd7be None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:15.687855 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6c369b-af50-433b-a73d-3b6b9c8dd7be None admin] PATH_INFO: `/v3/projects/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:15.694689 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6c369b-af50-433b-a73d-3b6b9c8dd7be None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:15.696137 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6c369b-af50-433b-a73d-3b6b9c8dd7be None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:15.708600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-cf6c369b-af50-433b-a73d-3b6b9c8dd7be None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:15.708600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:15.708600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:15.708600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:15.708600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:15.708600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:15.708600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:15.708600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:15.708600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:15.708600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:15.708600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:15.708600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:15.708600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:15.708600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:15.708600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:15.709475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:15.710455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:15.710455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:15.710455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:15.710455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:15.710455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:15.710455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:15.710455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:15.710455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:15.710455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 409/808] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:15.710455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6c369b-af50-433b-a73d-3b6b9c8dd7be None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:15.718833 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e92bc3-8167-4e2d-a38d-aa103f7dd271 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:15.753371 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6f2585-4ccc-424e-b509-ba013e8f2688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:15.753995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6f2585-4ccc-424e-b509-ba013e8f2688 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:15.754299 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6f2585-4ccc-424e-b509-ba013e8f2688 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:15.754569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6f2585-4ccc-424e-b509-ba013e8f2688 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:15.755251 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6f2585-4ccc-424e-b509-ba013e8f2688 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:15.756591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6f2585-4ccc-424e-b509-ba013e8f2688 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-3c6f2585-4ccc-424e-b509-ba013e8f2688 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:15.765887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:15.767063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:15.767063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:15.767063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:15.767063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:15.767063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:15.767063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:15.767063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:15.767063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:15.767063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:15.767063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:15.767063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:15.767063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:15.767063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:15.767063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:15.767063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:15.767063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:15.767896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:15.767896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:15.767896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:15.767896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:15.767896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:15.767896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:15.767896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:15.767896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:15.767896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:15.767896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:15.767896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:15.767896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:15.767896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 07 03:48:15.767896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:15.767896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 400/809] 10.222.0.27 () {56 vars in 1129 bytes} [Wed Sep 7 03:48:15 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:15.769176 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6f2585-4ccc-424e-b509-ba013e8f2688 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:15.820423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6c369b-af50-433b-a73d-3b6b9c8dd7be None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:15.828616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:15.854365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a61aab5-644a-407c-9cf9-47a3938f79bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:15.854993 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a61aab5-644a-407c-9cf9-47a3938f79bb None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:15.855270 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a61aab5-644a-407c-9cf9-47a3938f79bb None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:15.855553 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a61aab5-644a-407c-9cf9-47a3938f79bb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:15.862726 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a61aab5-644a-407c-9cf9-47a3938f79bb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:15.864173 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a61aab5-644a-407c-9cf9-47a3938f79bb None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:15.870583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6f2585-4ccc-424e-b509-ba013e8f2688 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:15.875308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-5a61aab5-644a-407c-9cf9-47a3938f79bb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:15.875308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:15.875308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:15.875308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:15.875308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:15.875308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:15.875308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:15.875308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:15.875308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:15.875308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:15.875308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:15.875308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:15.875308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:15.875308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:15.875308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:15.881212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:15.882375 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:15.882375 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:15.882375 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:15.882375 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:15.882375 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:15.882375 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:15.882375 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:15.882375 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:15.882375 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 410/810] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:15.882375 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a61aab5-644a-407c-9cf9-47a3938f79bb None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:15.903873 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06aaf953-12b7-4d50-9bdc-571f829fc2ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:15.904612 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06aaf953-12b7-4d50-9bdc-571f829fc2ca None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:15.904890 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06aaf953-12b7-4d50-9bdc-571f829fc2ca None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:15.905165 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06aaf953-12b7-4d50-9bdc-571f829fc2ca None admin] PATH_INFO: `/v3/roles/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:15.915505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06aaf953-12b7-4d50-9bdc-571f829fc2ca None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:15.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-06aaf953-12b7-4d50-9bdc-571f829fc2ca None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:48:15.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:15.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:15.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:15.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:15.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:15.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:15.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:15.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:15.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:15.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:15.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:15.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:15.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:15.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:15.921046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:15.922180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:15.922180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:15.922180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:15.922180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:15.922180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:15.922180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:15.922180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:15.922180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:15.922180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:15.922180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:15.922180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:15.922180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:15.922180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:15.922180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:15.922180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:15.922180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:15.923025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:15.923025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:15.923025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:15.923025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:15.923025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:15.923025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:15.923025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:15.923025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:15.923025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:15.923025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:15.923025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:48:15.923025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:15.923025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 401/811] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:48:15 2022] GET /identity/v3/roles/service => generated 85 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:15.923025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06aaf953-12b7-4d50-9bdc-571f829fc2ca None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:15.925631 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:15.986384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a61aab5-644a-407c-9cf9-47a3938f79bb None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:16.020911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61de900-cb2c-49e6-8af5-9018229997d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:16.021104 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06aaf953-12b7-4d50-9bdc-571f829fc2ca None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:16.021514 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61de900-cb2c-49e6-8af5-9018229997d9 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:16.021846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61de900-cb2c-49e6-8af5-9018229997d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:16.022134 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61de900-cb2c-49e6-8af5-9018229997d9 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:16.022849 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61de900-cb2c-49e6-8af5-9018229997d9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:16.023158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61de900-cb2c-49e6-8af5-9018229997d9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:16.024587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61de900-cb2c-49e6-8af5-9018229997d9 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:16.037225 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 411/812] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:16.037876 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:16.038014 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61de900-cb2c-49e6-8af5-9018229997d9 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:16.055967 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03918a21-e6fd-48e3-a6c4-9342adfe72cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:16.056568 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03918a21-e6fd-48e3-a6c4-9342adfe72cd None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:16.056843 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03918a21-e6fd-48e3-a6c4-9342adfe72cd None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:16.057109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03918a21-e6fd-48e3-a6c4-9342adfe72cd None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:16.057734 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03918a21-e6fd-48e3-a6c4-9342adfe72cd None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:16.058030 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03918a21-e6fd-48e3-a6c4-9342adfe72cd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:16.059409 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03918a21-e6fd-48e3-a6c4-9342adfe72cd None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:16.064222 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 402/813] 10.222.0.27 () {56 vars in 1124 bytes} [Wed Sep 7 03:48:15 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:16.065013 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03918a21-e6fd-48e3-a6c4-9342adfe72cd None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:16.139941 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61de900-cb2c-49e6-8af5-9018229997d9 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:16.141802 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:16.166084 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03918a21-e6fd-48e3-a6c4-9342adfe72cd None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:16.174064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde3f7c3-7809-4eb7-a825-8d222f402c6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:16.174716 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde3f7c3-7809-4eb7-a825-8d222f402c6e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:16.175001 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde3f7c3-7809-4eb7-a825-8d222f402c6e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:16.175266 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde3f7c3-7809-4eb7-a825-8d222f402c6e None admin] PATH_INFO: `/v3/services/network` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:16.175837 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde3f7c3-7809-4eb7-a825-8d222f402c6e None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:16.177197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde3f7c3-7809-4eb7-a825-8d222f402c6e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-cde3f7c3-7809-4eb7-a825-8d222f402c6e None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:16.185721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:16.191670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:16.191670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:16.191670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:16.191670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:16.191670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:16.191670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:16.191670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:16.191670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:16.191670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:16.191670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:16.191670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:16.191670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:16.191670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:16.191670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:16.191670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:16.191670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:16.192699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:16.192699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:16.192699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:16.192699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:16.192699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:16.192699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:16.192699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:16.192699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:16.192699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:16.192699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:16.192699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:16.192699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:16.192699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 07 03:48:16.192699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:16.192699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 412/814] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:16 2022] GET /identity/v3/services/network => generated 88 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:16.193662 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde3f7c3-7809-4eb7-a825-8d222f402c6e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:16.201082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed4865f-6c97-48e9-a71f-59b0644a7dae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:16.201720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed4865f-6c97-48e9-a71f-59b0644a7dae None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:16.202000 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed4865f-6c97-48e9-a71f-59b0644a7dae None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:16.202303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed4865f-6c97-48e9-a71f-59b0644a7dae None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:16.203003 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed4865f-6c97-48e9-a71f-59b0644a7dae None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:16.203307 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed4865f-6c97-48e9-a71f-59b0644a7dae None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:16.204986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed4865f-6c97-48e9-a71f-59b0644a7dae None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:16.217811 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 403/815] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:16.218639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed4865f-6c97-48e9-a71f-59b0644a7dae None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:16.263155 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:16.292837 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde3f7c3-7809-4eb7-a825-8d222f402c6e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:16.320867 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed4865f-6c97-48e9-a71f-59b0644a7dae None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:16.327030 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10f96f3-d1dc-4415-8369-3f2a349e3562 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:16.327643 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10f96f3-d1dc-4415-8369-3f2a349e3562 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:16.327924 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10f96f3-d1dc-4415-8369-3f2a349e3562 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:16.328197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10f96f3-d1dc-4415-8369-3f2a349e3562 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:16.329116 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10f96f3-d1dc-4415-8369-3f2a349e3562 None admin] RBAC: Adding query filter params (name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:16.329419 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10f96f3-d1dc-4415-8369-3f2a349e3562 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:16.330831 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10f96f3-d1dc-4415-8369-3f2a349e3562 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:16.338653 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 413/816] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:48:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:16.339437 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10f96f3-d1dc-4415-8369-3f2a349e3562 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:16.355437 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52157fd5-9d84-44f6-8897-35b9e7aa7dba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:16.356028 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52157fd5-9d84-44f6-8897-35b9e7aa7dba None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:16.356300 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52157fd5-9d84-44f6-8897-35b9e7aa7dba None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:16.356603 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52157fd5-9d84-44f6-8897-35b9e7aa7dba None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:16.357680 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52157fd5-9d84-44f6-8897-35b9e7aa7dba None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=953cb6c24bcc4551b91ce540d18cd751) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:16.357988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52157fd5-9d84-44f6-8897-35b9e7aa7dba None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:16.359424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52157fd5-9d84-44f6-8897-35b9e7aa7dba None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:16.372878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 404/817] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:48:16 2022] GET /identity/v3/role_assignments?user.id=953cb6c24bcc4551b91ce540d18cd751&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:16.372878 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52157fd5-9d84-44f6-8897-35b9e7aa7dba None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:16.375644 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:16.455666 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10f96f3-d1dc-4415-8369-3f2a349e3562 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:16.486431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52157fd5-9d84-44f6-8897-35b9e7aa7dba None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:16.493964 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:16.497682 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70661738-a931-42e6-b799-f07009be6f81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:16.498078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70661738-a931-42e6-b799-f07009be6f81 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:16.498402 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70661738-a931-42e6-b799-f07009be6f81 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:16.498679 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70661738-a931-42e6-b799-f07009be6f81 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:16.499474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70661738-a931-42e6-b799-f07009be6f81 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:16.499606 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70661738-a931-42e6-b799-f07009be6f81 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:16.501331 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70661738-a931-42e6-b799-f07009be6f81 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:16.512383 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 414/818] 10.222.0.27 () {56 vars in 1124 bytes} [Wed Sep 7 03:48:16 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:16.513934 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70661738-a931-42e6-b799-f07009be6f81 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:16.526490 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af194bbb-7384-446c-9b2c-58e31f93165e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:16.527106 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af194bbb-7384-446c-9b2c-58e31f93165e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:16.527381 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af194bbb-7384-446c-9b2c-58e31f93165e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:16.527651 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af194bbb-7384-446c-9b2c-58e31f93165e None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:16.528335 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af194bbb-7384-446c-9b2c-58e31f93165e None admin] RBAC: Adding query filter params (name=network) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:16.528831 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af194bbb-7384-446c-9b2c-58e31f93165e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:16.530604 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af194bbb-7384-446c-9b2c-58e31f93165e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:16.535533 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 405/819] 10.222.0.27 () {56 vars in 1112 bytes} [Wed Sep 7 03:48:16 2022] GET /identity/v3/services?name=network => generated 124 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:16.536329 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af194bbb-7384-446c-9b2c-58e31f93165e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:16.627516 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:16.639042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70661738-a931-42e6-b799-f07009be6f81 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:16.659642 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af194bbb-7384-446c-9b2c-58e31f93165e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:16.674332 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7b599d-e89e-4526-b0cf-b27553cc6e92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:16.674994 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7b599d-e89e-4526-b0cf-b27553cc6e92 None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:16.675270 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7b599d-e89e-4526-b0cf-b27553cc6e92 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:16.675543 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7b599d-e89e-4526-b0cf-b27553cc6e92 None admin] PATH_INFO: `/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/f5c3ab9728f74cd8bb2614bcca9e8619/roles/9d5b19a61c46475d8048b1bd66ca25a1` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:16.677453 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7b599d-e89e-4526-b0cf-b27553cc6e92 None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4816e95144485e8135e5b8f68c3662, user_id=f5c3ab9728f74cd8bb2614bcca9e8619, role_id=9d5b19a61c46475d8048b1bd66ca25a1)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:16.678928 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7b599d-e89e-4526-b0cf-b27553cc6e92 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:16.692316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 415/820] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:48:16 2022] PUT /identity/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/f5c3ab9728f74cd8bb2614bcca9e8619/roles/9d5b19a61c46475d8048b1bd66ca25a1 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:48:16.693106 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7b599d-e89e-4526-b0cf-b27553cc6e92 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:16.710436 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cdafd6-946a-4b2b-be73-95d76add4df2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:16.711188 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cdafd6-946a-4b2b-be73-95d76add4df2 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:16.711582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cdafd6-946a-4b2b-be73-95d76add4df2 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:16.711932 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cdafd6-946a-4b2b-be73-95d76add4df2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:16.725632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cdafd6-946a-4b2b-be73-95d76add4df2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:16.727386 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cdafd6-946a-4b2b-be73-95d76add4df2 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:16.745942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-65cdafd6-946a-4b2b-be73-95d76add4df2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:16.745942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:16.745942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:16.745942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:16.745942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:16.745942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:16.745942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:16.745942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:16.745942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:16.745942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:16.745942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:16.745942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:16.745942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:16.745942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:16.745942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:16.745942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:16.746985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:16.749380 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:16.749380 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:16.749380 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:16.749380 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 406/821] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:16.749868 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cdafd6-946a-4b2b-be73-95d76add4df2 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:16.779412 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:16.820807 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7b599d-e89e-4526-b0cf-b27553cc6e92 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:16.866234 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe215e4-b458-4dc6-8dd3-9fa4dc9e9937 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:16.867251 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe215e4-b458-4dc6-8dd3-9fa4dc9e9937 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:16.867744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe215e4-b458-4dc6-8dd3-9fa4dc9e9937 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:16.868259 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe215e4-b458-4dc6-8dd3-9fa4dc9e9937 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:16.869232 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe215e4-b458-4dc6-8dd3-9fa4dc9e9937 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:16.870964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe215e4-b458-4dc6-8dd3-9fa4dc9e9937 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:16.881365 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cdafd6-946a-4b2b-be73-95d76add4df2 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:16.899923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 416/822] 10.222.0.27 () {60 vars in 1144 bytes} [Wed Sep 7 03:48:16 2022] POST /identity/v3/endpoints => generated 346 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:48:16.901193 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe215e4-b458-4dc6-8dd3-9fa4dc9e9937 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:16.917005 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:16.949804 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc94eff-8dbc-4579-8868-27ae730963c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:16.950993 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc94eff-8dbc-4579-8868-27ae730963c8 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:16.951273 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc94eff-8dbc-4579-8868-27ae730963c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:16.951598 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc94eff-8dbc-4579-8868-27ae730963c8 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:16.952433 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc94eff-8dbc-4579-8868-27ae730963c8 None admin] RBAC: Adding query filter params (type=network) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:16.952719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc94eff-8dbc-4579-8868-27ae730963c8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:16.954844 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc94eff-8dbc-4579-8868-27ae730963c8 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:16.965676 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 407/823] 10.222.0.27 () {56 vars in 1112 bytes} [Wed Sep 7 03:48:16 2022] GET /identity/v3/services?type=network => generated 350 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:16.971346 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc94eff-8dbc-4579-8868-27ae730963c8 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:17.053463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe215e4-b458-4dc6-8dd3-9fa4dc9e9937 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:17.093174 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9b929f-e932-476f-abb8-a2551a1aaa18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:17.094267 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9b929f-e932-476f-abb8-a2551a1aaa18 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:17.094762 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9b929f-e932-476f-abb8-a2551a1aaa18 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:17.095266 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9b929f-e932-476f-abb8-a2551a1aaa18 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:17.096144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9b929f-e932-476f-abb8-a2551a1aaa18 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:17.096657 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9b929f-e932-476f-abb8-a2551a1aaa18 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:17.098367 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9b929f-e932-476f-abb8-a2551a1aaa18 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:17.101205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:17.114272 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 417/824] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:17.119415 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9b929f-e932-476f-abb8-a2551a1aaa18 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:17.142453 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc94eff-8dbc-4579-8868-27ae730963c8 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:17.193195 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09207cf7-50e5-4ceb-a886-4656faa9ba36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:17.193844 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09207cf7-50e5-4ceb-a886-4656faa9ba36 None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:17.194435 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:17.194670 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09207cf7-50e5-4ceb-a886-4656faa9ba36 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:17.195874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09207cf7-50e5-4ceb-a886-4656faa9ba36 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:17.196875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09207cf7-50e5-4ceb-a886-4656faa9ba36 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:17.199651 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09207cf7-50e5-4ceb-a886-4656faa9ba36 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:17.229416 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 408/825] 10.222.0.27 () {60 vars in 1144 bytes} [Wed Sep 7 03:48:16 2022] POST /identity/v3/endpoints => generated 337 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 07 03:48:17.239888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9b929f-e932-476f-abb8-a2551a1aaa18 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:17.286344 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a613f897-81c9-46eb-89e8-66b580fc3e1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:17.286996 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a613f897-81c9-46eb-89e8-66b580fc3e1d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:17.287272 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a613f897-81c9-46eb-89e8-66b580fc3e1d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:17.287591 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a613f897-81c9-46eb-89e8-66b580fc3e1d None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:17.296381 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a613f897-81c9-46eb-89e8-66b580fc3e1d None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:17.297971 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a613f897-81c9-46eb-89e8-66b580fc3e1d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:17.311880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-a613f897-81c9-46eb-89e8-66b580fc3e1d None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 07 03:48:17.311880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:17.311880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:17.311880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:17.311880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:17.311880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:17.311880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:17.311880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:17.311880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:17.311880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:17.311880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:17.311880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:17.311880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:17.311880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:17.311880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:17.311880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:17.312886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:17.313810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:17.313810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:17.313810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:17.313810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:17.313810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:17.313810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:17.313810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:17.313810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:17.313810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 07 03:48:17.313810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:17.313810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 418/826] 10.222.0.27 () {56 vars in 1117 bytes} [Wed Sep 7 03:48:17 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:17.319378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09207cf7-50e5-4ceb-a886-4656faa9ba36 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:17.441615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09207cf7-50e5-4ceb-a886-4656faa9ba36 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:17.478610 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3b130b-5b42-4e3f-95b6-4419201c8799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:17.479225 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3b130b-5b42-4e3f-95b6-4419201c8799 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:17.479600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3b130b-5b42-4e3f-95b6-4419201c8799 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:17.479822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3b130b-5b42-4e3f-95b6-4419201c8799 None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:17.480557 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3b130b-5b42-4e3f-95b6-4419201c8799 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:17.480859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3b130b-5b42-4e3f-95b6-4419201c8799 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:17.482411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3b130b-5b42-4e3f-95b6-4419201c8799 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:17.491020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:17.506316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 409/827] 10.222.0.27 () {56 vars in 1149 bytes} [Wed Sep 7 03:48:17 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:17.511939 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a613f897-81c9-46eb-89e8-66b580fc3e1d None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:17.626823 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a613f897-81c9-46eb-89e8-66b580fc3e1d None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:17.652003 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:17.664950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61003d3-8a17-4d1f-93cb-e035e486cf6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:17.665586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61003d3-8a17-4d1f-93cb-e035e486cf6b None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:17.665864 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61003d3-8a17-4d1f-93cb-e035e486cf6b None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:17.666155 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61003d3-8a17-4d1f-93cb-e035e486cf6b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:17.677664 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61003d3-8a17-4d1f-93cb-e035e486cf6b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:17.679170 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61003d3-8a17-4d1f-93cb-e035e486cf6b None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:17.696559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-d61003d3-8a17-4d1f-93cb-e035e486cf6b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:17.696559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:17.696559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:17.696559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:17.696559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:17.696559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:17.696559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:17.696559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:17.696559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:17.696559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:17.696559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:17.696559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:17.696559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:17.696559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:17.696559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:17.696559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:17.697512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:17.698458 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:17.698458 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:17.698458 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:17.698458 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 419/828] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:17.704799 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3b130b-5b42-4e3f-95b6-4419201c8799 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:17.818076 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3b130b-5b42-4e3f-95b6-4419201c8799 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:17.856830 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4def26-a792-4d57-85f6-7410b8320b38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:17.857516 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4def26-a792-4d57-85f6-7410b8320b38 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:17.857804 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4def26-a792-4d57-85f6-7410b8320b38 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:17.858081 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4def26-a792-4d57-85f6-7410b8320b38 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:17.858842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4def26-a792-4d57-85f6-7410b8320b38 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:17.859147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4def26-a792-4d57-85f6-7410b8320b38 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:17.860507 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4def26-a792-4d57-85f6-7410b8320b38 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:17.873105 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 410/829] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:17.878319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61003d3-8a17-4d1f-93cb-e035e486cf6b None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:17.988832 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61003d3-8a17-4d1f-93cb-e035e486cf6b None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:18.023595 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74108862-adf1-4521-a1d4-b489ed150d36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:18.024229 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74108862-adf1-4521-a1d4-b489ed150d36 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:18.024505 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74108862-adf1-4521-a1d4-b489ed150d36 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:18.024777 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74108862-adf1-4521-a1d4-b489ed150d36 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:18.032516 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74108862-adf1-4521-a1d4-b489ed150d36 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:18.033959 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74108862-adf1-4521-a1d4-b489ed150d36 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:18.046340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-74108862-adf1-4521-a1d4-b489ed150d36 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:18.046340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:18.046340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:18.046340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:18.046340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:18.046340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:18.046340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:18.046340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:18.046340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:18.046340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:18.046340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:18.046340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:18.046340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:18.046340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:18.046340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:18.047328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:18.048197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:18.048197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:18.048197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:18.048197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:18.048197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:18.048197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:18.048197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:18.048197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:18.048197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 420/830] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:18.053287 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4def26-a792-4d57-85f6-7410b8320b38 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:18.114019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:18.159082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4def26-a792-4d57-85f6-7410b8320b38 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:18.194601 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b698694b-6ef8-40bf-86c8-2a62603cbce3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:18.195215 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b698694b-6ef8-40bf-86c8-2a62603cbce3 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:18.195497 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b698694b-6ef8-40bf-86c8-2a62603cbce3 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:18.195764 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b698694b-6ef8-40bf-86c8-2a62603cbce3 None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:18.196452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b698694b-6ef8-40bf-86c8-2a62603cbce3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:18.196755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b698694b-6ef8-40bf-86c8-2a62603cbce3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:18.198185 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b698694b-6ef8-40bf-86c8-2a62603cbce3 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:18.211804 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 411/831] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:18.217824 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74108862-adf1-4521-a1d4-b489ed150d36 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:18.259808 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:18.332186 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74108862-adf1-4521-a1d4-b489ed150d36 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:18.367754 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e9644e-5d91-4772-a0e7-8bdcf0ada01f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:18.368391 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e9644e-5d91-4772-a0e7-8bdcf0ada01f None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:18.368675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e9644e-5d91-4772-a0e7-8bdcf0ada01f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:18.368947 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e9644e-5d91-4772-a0e7-8bdcf0ada01f None admin] PATH_INFO: `/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/e7b8f511ee884dbfb4f56af78bf75984/roles/a7b16bb097f14f78a7a31c04bd6d32bd` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:18.384099 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e9644e-5d91-4772-a0e7-8bdcf0ada01f None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4816e95144485e8135e5b8f68c3662, user_id=e7b8f511ee884dbfb4f56af78bf75984, role_id=a7b16bb097f14f78a7a31c04bd6d32bd)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:18.385546 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e9644e-5d91-4772-a0e7-8bdcf0ada01f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:18.400586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 421/832] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:48:18 2022] PUT /identity/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/e7b8f511ee884dbfb4f56af78bf75984/roles/a7b16bb097f14f78a7a31c04bd6d32bd => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:48:18.804546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823cc794-0430-4513-8518-8b5ca5fe14fb None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:18.805214 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823cc794-0430-4513-8518-8b5ca5fe14fb None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:18.805708 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823cc794-0430-4513-8518-8b5ca5fe14fb None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:18.807526 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 412/833] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:18.813458 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271edc10-060a-4631-8406-c87f78afdea6 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:18.814314 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271edc10-060a-4631-8406-c87f78afdea6 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:18.814803 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271edc10-060a-4631-8406-c87f78afdea6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:18.833504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-271edc10-060a-4631-8406-c87f78afdea6 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:18.835051 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-271edc10-060a-4631-8406-c87f78afdea6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:18.904657 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 422/834] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:18 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:19.173254 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823cc794-0430-4513-8518-8b5ca5fe14fb None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:19.181876 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb78e2f-5ba8-4bb0-b07d-1ca833d93a6a None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:19.182196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb78e2f-5ba8-4bb0-b07d-1ca833d93a6a None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:19.182482 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb78e2f-5ba8-4bb0-b07d-1ca833d93a6a None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:19.183978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 423/835] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:19.189769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898cb2bd-61af-462c-986d-61428b4b3450 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:19.190472 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898cb2bd-61af-462c-986d-61428b4b3450 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:19.190754 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898cb2bd-61af-462c-986d-61428b4b3450 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:19.212995 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-898cb2bd-61af-462c-986d-61428b4b3450 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:19.215093 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-898cb2bd-61af-462c-986d-61428b4b3450 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:19.229668 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:19.306148 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 424/836] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:19 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:48:19.315797 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823cc794-0430-4513-8518-8b5ca5fe14fb None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:19.336351 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:19.358728 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4574ad3-a36e-4411-9f14-da5e46b16aeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:19.359430 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4574ad3-a36e-4411-9f14-da5e46b16aeb None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:19.359711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4574ad3-a36e-4411-9f14-da5e46b16aeb None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:19.360029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4574ad3-a36e-4411-9f14-da5e46b16aeb None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:19.360625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4574ad3-a36e-4411-9f14-da5e46b16aeb None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:19.362043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4574ad3-a36e-4411-9f14-da5e46b16aeb None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:19.371493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-c4574ad3-a36e-4411-9f14-da5e46b16aeb None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 07 03:48:19.371493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:19.371493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:19.371493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:19.371493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:19.371493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:19.371493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:19.371493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:19.371493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 07 03:48:19.371493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:19.371493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:19.371493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:19.371493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:19.371493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:19.371493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:19.371493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:19.373190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:19.374653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:19.374653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:19.374653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:19.374653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:19.374653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:19.374653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:19.374653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:19.374653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:19.374653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:19.374653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:19.374653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:19.374653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:19.374653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 07 03:48:19.374653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:19.374653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 413/837] 10.222.0.27 () {56 vars in 1132 bytes} [Wed Sep 7 03:48:19 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:19.379094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898cb2bd-61af-462c-986d-61428b4b3450 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:19.483624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898cb2bd-61af-462c-986d-61428b4b3450 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:19.519643 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7d6481-109c-4c6a-be55-7b179a310a4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:19.520258 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7d6481-109c-4c6a-be55-7b179a310a4b None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:19.520540 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7d6481-109c-4c6a-be55-7b179a310a4b None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:19.520814 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7d6481-109c-4c6a-be55-7b179a310a4b None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:19.521447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7d6481-109c-4c6a-be55-7b179a310a4b None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:19.521749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7d6481-109c-4c6a-be55-7b179a310a4b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:19.523148 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7d6481-109c-4c6a-be55-7b179a310a4b None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:19.527982 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 425/838] 10.222.0.27 () {56 vars in 1126 bytes} [Wed Sep 7 03:48:19 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:19.533403 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4574ad3-a36e-4411-9f14-da5e46b16aeb None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:19.597591 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3fdc70-3898-4f29-b37f-2f2e0232e155 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:19.597914 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3fdc70-3898-4f29-b37f-2f2e0232e155 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:19.598221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3fdc70-3898-4f29-b37f-2f2e0232e155 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:19.600628 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 426/839] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:48:19.603395 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3fdc70-3898-4f29-b37f-2f2e0232e155 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:19.615805 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:19.645009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4574ad3-a36e-4411-9f14-da5e46b16aeb None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:19.679633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa74ae6-58dc-4545-9569-78fc917c5e8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:19.680280 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa74ae6-58dc-4545-9569-78fc917c5e8d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:19.680564 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa74ae6-58dc-4545-9569-78fc917c5e8d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:19.680850 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa74ae6-58dc-4545-9569-78fc917c5e8d None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:19.681487 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa74ae6-58dc-4545-9569-78fc917c5e8d None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:19.681799 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa74ae6-58dc-4545-9569-78fc917c5e8d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:19.683202 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa74ae6-58dc-4545-9569-78fc917c5e8d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:19.694240 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 414/840] 10.222.0.27 () {56 vars in 1126 bytes} [Wed Sep 7 03:48:19 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:19.694240 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f536ddf0-78cf-495e-9a86-af4461d5d36b None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:19.694240 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f536ddf0-78cf-495e-9a86-af4461d5d36b None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:19.694240 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f536ddf0-78cf-495e-9a86-af4461d5d36b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:19.712214 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-f536ddf0-78cf-495e-9a86-af4461d5d36b None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:19.713815 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-f536ddf0-78cf-495e-9a86-af4461d5d36b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:19.726857 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3fdc70-3898-4f29-b37f-2f2e0232e155 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:19.764261 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:19.781590 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56a4887-0d60-4beb-921b-c62f3f60d902 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:19.782433 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56a4887-0d60-4beb-921b-c62f3f60d902 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:19.782805 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56a4887-0d60-4beb-921b-c62f3f60d902 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:19.783157 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56a4887-0d60-4beb-921b-c62f3f60d902 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:19.799614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56a4887-0d60-4beb-921b-c62f3f60d902 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:19.805966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-a56a4887-0d60-4beb-921b-c62f3f60d902 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:19.805966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:19.805966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:19.805966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:19.805966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:19.805966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:19.805966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:19.805966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:19.805966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:19.805966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:19.805966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:19.805966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:19.805966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:19.805966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:19.805966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:19.805966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:19.807373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:19.807373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:19.807373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:19.807373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:19.807373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:19.807373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:19.807373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:19.807373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:19.807373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:19.807373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:19.807373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:19.807373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:19.807373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:19.807373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:19.807373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:19.807373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:19.808260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:19.808260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:19.808260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:19.808260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:19.808260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:19.808260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:19.808260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:19.808260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:19.808260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:19.808260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:19.808260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:19.808260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:19.808260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 427/841] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:48:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:19.809524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 415/842] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:19 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:19.816083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56a4887-0d60-4beb-921b-c62f3f60d902 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:19.900273 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:19.933987 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56a4887-0d60-4beb-921b-c62f3f60d902 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:19.970929 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9080796e-4962-4416-8424-ee06579d5c04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:19.971645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9080796e-4962-4416-8424-ee06579d5c04 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:19.971938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9080796e-4962-4416-8424-ee06579d5c04 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:19.972215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9080796e-4962-4416-8424-ee06579d5c04 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:19.972889 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9080796e-4962-4416-8424-ee06579d5c04 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:19.973196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9080796e-4962-4416-8424-ee06579d5c04 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:19.974599 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9080796e-4962-4416-8424-ee06579d5c04 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:19.984012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 428/843] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:48:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:19.989842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f536ddf0-78cf-495e-9a86-af4461d5d36b None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:20.094475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9080796e-4962-4416-8424-ee06579d5c04 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:20.108789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f536ddf0-78cf-495e-9a86-af4461d5d36b None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:20.145015 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4a6d74-5bac-46e0-9c48-f6a38dea0d3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:20.145688 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4a6d74-5bac-46e0-9c48-f6a38dea0d3b None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:20.145974 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4a6d74-5bac-46e0-9c48-f6a38dea0d3b None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:20.146279 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4a6d74-5bac-46e0-9c48-f6a38dea0d3b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:20.157479 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4a6d74-5bac-46e0-9c48-f6a38dea0d3b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:20.159012 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4a6d74-5bac-46e0-9c48-f6a38dea0d3b None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:20.175552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-ec4a6d74-5bac-46e0-9c48-f6a38dea0d3b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:20.175552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:20.175552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:20.175552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:20.175552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:20.175552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:20.175552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:20.175552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:20.175552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:20.175552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:20.175552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:20.175552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:20.175552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:20.175552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:20.175552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:20.175552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:20.177035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:20.177937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:20.177937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:20.177937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:20.181478 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 416/844] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:20.182829 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4a6d74-5bac-46e0-9c48-f6a38dea0d3b None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:20.219929 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9080796e-4962-4416-8424-ee06579d5c04 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:20.224283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:20.267196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:20.275245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621d02e3-ee5c-4c61-96b3-f762635f12ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:20.276342 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621d02e3-ee5c-4c61-96b3-f762635f12ec None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:20.276788 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621d02e3-ee5c-4c61-96b3-f762635f12ec None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:20.277680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621d02e3-ee5c-4c61-96b3-f762635f12ec None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:20.278924 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621d02e3-ee5c-4c61-96b3-f762635f12ec None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:20.280617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621d02e3-ee5c-4c61-96b3-f762635f12ec None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-621d02e3-ee5c-4c61-96b3-f762635f12ec None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:20.293294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:20.294615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:20.294615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:20.294615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:20.294615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:20.294615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:20.294615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:20.294615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:20.294615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:20.294615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:20.294615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:20.294615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:20.294615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:20.294615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:20.294615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:20.294615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:20.294615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:20.295604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:20.295604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:20.295604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:20.295604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:20.295604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:20.295604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:20.295604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:20.295604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:20.295604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:20.295604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:20.295604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:20.295604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:20.295604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 07 03:48:20.295604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:20.298758 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 429/845] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:48:20 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:20.304414 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621d02e3-ee5c-4c61-96b3-f762635f12ec None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:20.321068 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4a6d74-5bac-46e0-9c48-f6a38dea0d3b None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:20.364919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486ecd26-f01f-44a5-992d-abdc4cc2000c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:20.365864 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486ecd26-f01f-44a5-992d-abdc4cc2000c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:20.366453 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486ecd26-f01f-44a5-992d-abdc4cc2000c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:20.366936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486ecd26-f01f-44a5-992d-abdc4cc2000c None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:20.367938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486ecd26-f01f-44a5-992d-abdc4cc2000c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:20.368461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486ecd26-f01f-44a5-992d-abdc4cc2000c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:20.370021 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486ecd26-f01f-44a5-992d-abdc4cc2000c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:20.389071 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 417/846] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:20.390454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486ecd26-f01f-44a5-992d-abdc4cc2000c None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:20.426326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621d02e3-ee5c-4c61-96b3-f762635f12ec None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:20.471758 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909b755f-ef2a-4b8d-b22f-b144dc3e4f0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:20.472380 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909b755f-ef2a-4b8d-b22f-b144dc3e4f0f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:20.472662 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909b755f-ef2a-4b8d-b22f-b144dc3e4f0f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:20.472937 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909b755f-ef2a-4b8d-b22f-b144dc3e4f0f None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:20.473576 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909b755f-ef2a-4b8d-b22f-b144dc3e4f0f None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:20.473869 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909b755f-ef2a-4b8d-b22f-b144dc3e4f0f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:20.475991 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909b755f-ef2a-4b8d-b22f-b144dc3e4f0f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:20.486217 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 430/847] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:48:20 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:20.487934 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:20.489449 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909b755f-ef2a-4b8d-b22f-b144dc3e4f0f None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:20.505079 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486ecd26-f01f-44a5-992d-abdc4cc2000c None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:20.543362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82020ab-540b-47a7-b209-f215a2f477e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:20.544543 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82020ab-540b-47a7-b209-f215a2f477e1 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:20.545079 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82020ab-540b-47a7-b209-f215a2f477e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:20.545588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82020ab-540b-47a7-b209-f215a2f477e1 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:20.554581 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82020ab-540b-47a7-b209-f215a2f477e1 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:20.556703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82020ab-540b-47a7-b209-f215a2f477e1 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:20.569359 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:20.572233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-a82020ab-540b-47a7-b209-f215a2f477e1 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 07 03:48:20.572233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:20.572233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:20.572233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:20.572233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:20.572233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:20.572233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:20.572233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:20.572233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:20.572233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:20.572233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:20.572233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:20.572233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:20.572233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:20.572233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:20.572233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:20.573209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:20.574173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:20.574173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:20.574173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:20.574173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:20.574173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:20.574173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:20.574173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:20.574173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:20.574173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 07 03:48:20.574173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:20.582433 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 418/848] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:48:20 2022] GET /identity/v3/users/swift => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:20.584203 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82020ab-540b-47a7-b209-f215a2f477e1 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:20.619440 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909b755f-ef2a-4b8d-b22f-b144dc3e4f0f None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:20.660147 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fced7354-9aad-43c9-9aea-fee7ef475da6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:20.661171 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fced7354-9aad-43c9-9aea-fee7ef475da6 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:20.661677 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fced7354-9aad-43c9-9aea-fee7ef475da6 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:20.662264 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fced7354-9aad-43c9-9aea-fee7ef475da6 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:20.663283 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fced7354-9aad-43c9-9aea-fee7ef475da6 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:20.664043 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fced7354-9aad-43c9-9aea-fee7ef475da6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:20.665722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fced7354-9aad-43c9-9aea-fee7ef475da6 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:20.676594 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 431/849] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:48:20 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:20.676594 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6c75fa-431e-45c6-9cf4-487b575dea59 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:20.676594 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6c75fa-431e-45c6-9cf4-487b575dea59 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:20.676594 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6c75fa-431e-45c6-9cf4-487b575dea59 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:20.679107 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 432/850] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:20.683659 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d477866d-3389-4a28-9362-18636edfeed3 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:20.683970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d477866d-3389-4a28-9362-18636edfeed3 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:20.684214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d477866d-3389-4a28-9362-18636edfeed3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:20.705810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-d477866d-3389-4a28-9362-18636edfeed3 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:20.707178 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-d477866d-3389-4a28-9362-18636edfeed3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:20.719213 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82020ab-540b-47a7-b209-f215a2f477e1 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:20.769586 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90a7884-3059-49af-9bca-e39192552219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:20.770227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90a7884-3059-49af-9bca-e39192552219 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:20.770508 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90a7884-3059-49af-9bca-e39192552219 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:20.771488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90a7884-3059-49af-9bca-e39192552219 None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:20.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90a7884-3059-49af-9bca-e39192552219 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:20.772857 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90a7884-3059-49af-9bca-e39192552219 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:20.775424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90a7884-3059-49af-9bca-e39192552219 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:20.794959 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 433/851] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:20 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:48:20.797638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 419/852] 10.222.0.27 () {56 vars in 1135 bytes} [Wed Sep 7 03:48:20 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:20.803278 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d477866d-3389-4a28-9362-18636edfeed3 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:20.873172 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:20.922900 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d477866d-3389-4a28-9362-18636edfeed3 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:20.959422 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3c220f-4235-4b99-b4ba-5ecf96647403 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:20.960460 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3c220f-4235-4b99-b4ba-5ecf96647403 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:20.960967 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3c220f-4235-4b99-b4ba-5ecf96647403 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:20.961448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3c220f-4235-4b99-b4ba-5ecf96647403 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:21.034764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3c220f-4235-4b99-b4ba-5ecf96647403 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:21.036238 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3c220f-4235-4b99-b4ba-5ecf96647403 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:21.059266 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90a7884-3059-49af-9bca-e39192552219 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:21.065666 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:21.098495 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:21.149104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 434/853] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:48:20 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:48:21.150313 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3c220f-4235-4b99-b4ba-5ecf96647403 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:21.170451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90a7884-3059-49af-9bca-e39192552219 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:21.208804 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e4bf33-1ed8-49c5-b5ad-53d9a7c5b99a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:21.210244 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4bf33-1ed8-49c5-b5ad-53d9a7c5b99a None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:21.210723 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4bf33-1ed8-49c5-b5ad-53d9a7c5b99a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:21.211208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4bf33-1ed8-49c5-b5ad-53d9a7c5b99a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:21.223350 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e4bf33-1ed8-49c5-b5ad-53d9a7c5b99a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:21.229571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-79e4bf33-1ed8-49c5-b5ad-53d9a7c5b99a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:48:21.229571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:21.229571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:21.229571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:21.229571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:21.229571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:21.229571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:21.229571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:21.229571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:21.229571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:21.229571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:21.229571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:21.229571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:21.229571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:21.229571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:21.229571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:21.230529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:21.230529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:21.230529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:21.230529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:21.230529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:21.230529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:21.230529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:21.230529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:21.230529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:21.230529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:21.230529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:21.230529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:21.230529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:21.230529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:21.230529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:21.230529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:21.231525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:21.231525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:21.231525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:21.231525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:21.231525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:21.231525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:21.231525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:21.231525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:21.231525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:21.231525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:21.231525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:48:21.231525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:21.235053 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 420/854] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:48:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:21.240109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e4bf33-1ed8-49c5-b5ad-53d9a7c5b99a None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:21.269182 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3c220f-4235-4b99-b4ba-5ecf96647403 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:21.306192 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:21.307978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f367e3b-8e81-4364-a3d9-15475067545f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:21.308859 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f367e3b-8e81-4364-a3d9-15475067545f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:21.309441 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f367e3b-8e81-4364-a3d9-15475067545f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:21.310286 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f367e3b-8e81-4364-a3d9-15475067545f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:21.322316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f367e3b-8e81-4364-a3d9-15475067545f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:21.324118 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f367e3b-8e81-4364-a3d9-15475067545f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:21.341794 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-6f367e3b-8e81-4364-a3d9-15475067545f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:21.341794 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:21.341794 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:21.341794 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:21.341794 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:21.341794 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:21.341794 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:21.341794 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:21.341794 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:21.341794 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:21.341794 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:21.341794 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:21.341794 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:21.341794 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:21.341794 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:21.341794 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:21.342795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:21.343668 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:21.343668 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:21.343668 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:21.345737 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 435/855] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:21.345920 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e4bf33-1ed8-49c5-b5ad-53d9a7c5b99a None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:21.351054 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f367e3b-8e81-4364-a3d9-15475067545f None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:21.361844 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:21.389870 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccdfc02-e99d-4c98-9a5e-3f8ba4ddbfd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:21.390676 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccdfc02-e99d-4c98-9a5e-3f8ba4ddbfd6 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:21.391294 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccdfc02-e99d-4c98-9a5e-3f8ba4ddbfd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:21.391667 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccdfc02-e99d-4c98-9a5e-3f8ba4ddbfd6 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:21.392838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccdfc02-e99d-4c98-9a5e-3f8ba4ddbfd6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:21.393391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccdfc02-e99d-4c98-9a5e-3f8ba4ddbfd6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:21.394923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccdfc02-e99d-4c98-9a5e-3f8ba4ddbfd6 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:21.410231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 421/856] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:48:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:21.412715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccdfc02-e99d-4c98-9a5e-3f8ba4ddbfd6 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:21.475391 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f367e3b-8e81-4364-a3d9-15475067545f None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:21.512008 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6ccd60-9863-42bc-8577-6814a0f33d75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:21.513181 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6ccd60-9863-42bc-8577-6814a0f33d75 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:21.513603 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6ccd60-9863-42bc-8577-6814a0f33d75 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:21.514090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6ccd60-9863-42bc-8577-6814a0f33d75 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:21.514799 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6ccd60-9863-42bc-8577-6814a0f33d75 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:21.515104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6ccd60-9863-42bc-8577-6814a0f33d75 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:21.516685 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6ccd60-9863-42bc-8577-6814a0f33d75 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:21.529913 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 436/857] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:21.535379 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6ccd60-9863-42bc-8577-6814a0f33d75 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:21.546547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccdfc02-e99d-4c98-9a5e-3f8ba4ddbfd6 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:21.581808 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:21.592433 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d540c243-935c-4911-9193-bebcafa56642 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:21.592937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d540c243-935c-4911-9193-bebcafa56642 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:21.593222 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d540c243-935c-4911-9193-bebcafa56642 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:21.593501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d540c243-935c-4911-9193-bebcafa56642 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:21.604802 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d540c243-935c-4911-9193-bebcafa56642 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:21.606340 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d540c243-935c-4911-9193-bebcafa56642 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:21.618501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:21.621696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-d540c243-935c-4911-9193-bebcafa56642 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:21.621696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:21.621696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:21.621696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:21.621696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:21.621696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:21.621696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:21.621696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:21.621696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:21.621696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:21.621696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:21.621696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:21.621696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:21.621696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:21.621696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:21.621696 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:21.623411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:21.624304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:21.624304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:21.624304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:21.628495 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 422/858] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:21.629976 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d540c243-935c-4911-9193-bebcafa56642 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:21.661939 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6ccd60-9863-42bc-8577-6814a0f33d75 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:21.701357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527df686-d072-46bc-b6f8-9cedd682a881 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:21.702304 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527df686-d072-46bc-b6f8-9cedd682a881 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:21.702729 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527df686-d072-46bc-b6f8-9cedd682a881 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:21.703145 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527df686-d072-46bc-b6f8-9cedd682a881 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:21.710169 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527df686-d072-46bc-b6f8-9cedd682a881 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:21.711777 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527df686-d072-46bc-b6f8-9cedd682a881 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:21.723242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-527df686-d072-46bc-b6f8-9cedd682a881 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:21.723242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:21.723242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:21.723242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:21.723242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:21.723242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:21.723242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:21.723242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:21.723242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:21.723242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:21.723242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:21.723242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:21.723242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:21.723242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:21.723242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:21.724306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:21.725292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:21.725292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:21.725292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:21.725292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:21.725292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:21.725292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:21.725292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:21.725292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:21.727294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 437/859] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:21.733736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527df686-d072-46bc-b6f8-9cedd682a881 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:21.749030 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d540c243-935c-4911-9193-bebcafa56642 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:21.786053 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4700f8-ce15-474d-8954-813624df5959 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:21.787077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4700f8-ce15-474d-8954-813624df5959 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:21.787554 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4700f8-ce15-474d-8954-813624df5959 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:21.788022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4700f8-ce15-474d-8954-813624df5959 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:21.788912 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4700f8-ce15-474d-8954-813624df5959 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:21.789420 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4700f8-ce15-474d-8954-813624df5959 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:21.791034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4700f8-ce15-474d-8954-813624df5959 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:21.805085 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 423/860] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:21.811965 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4700f8-ce15-474d-8954-813624df5959 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:21.859638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527df686-d072-46bc-b6f8-9cedd682a881 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:21.868171 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:21.902505 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:21.915467 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afad1d67-a254-4401-898a-06e207ba3c6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:21.916883 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afad1d67-a254-4401-898a-06e207ba3c6f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:21.917495 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afad1d67-a254-4401-898a-06e207ba3c6f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:21.918456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afad1d67-a254-4401-898a-06e207ba3c6f None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:21.919870 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afad1d67-a254-4401-898a-06e207ba3c6f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:21.920425 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afad1d67-a254-4401-898a-06e207ba3c6f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:21.922479 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afad1d67-a254-4401-898a-06e207ba3c6f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:21.937041 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4700f8-ce15-474d-8954-813624df5959 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:21.946432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 438/861] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:21.948892 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afad1d67-a254-4401-898a-06e207ba3c6f None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:21.976912 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8546f2a0-6f50-43ae-83c3-68af2cd80362 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:21.977542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8546f2a0-6f50-43ae-83c3-68af2cd80362 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:21.977819 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8546f2a0-6f50-43ae-83c3-68af2cd80362 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:21.978092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8546f2a0-6f50-43ae-83c3-68af2cd80362 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:21.986033 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8546f2a0-6f50-43ae-83c3-68af2cd80362 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:21.987503 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8546f2a0-6f50-43ae-83c3-68af2cd80362 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:21.999506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-8546f2a0-6f50-43ae-83c3-68af2cd80362 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 07 03:48:21.999506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:21.999506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:21.999506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:21.999506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:21.999506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:21.999506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:21.999506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:21.999506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:21.999506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:21.999506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:21.999506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:21.999506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:21.999506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:21.999506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:21.999506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:22.000468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:22.001410 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:22.001410 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:22.001410 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:22.001410 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:22.001410 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:22.001410 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:22.001410 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:22.001410 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:22.001410 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 07 03:48:22.001410 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:22.001410 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 424/862] 10.222.0.27 () {56 vars in 1117 bytes} [Wed Sep 7 03:48:21 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:22.001999 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b916642f-5aa9-4301-ab03-8bcc45069f1d None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:22.002067 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b916642f-5aa9-4301-ab03-8bcc45069f1d None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:22.002703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b916642f-5aa9-4301-ab03-8bcc45069f1d None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:22.003747 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 425/863] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:22 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:22.008324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b916642f-5aa9-4301-ab03-8bcc45069f1d None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:22.065875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afad1d67-a254-4401-898a-06e207ba3c6f None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:22.101238 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fcd970-ec02-462e-8764-58b8b6c4624a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:22.101840 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fcd970-ec02-462e-8764-58b8b6c4624a None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:22.102146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fcd970-ec02-462e-8764-58b8b6c4624a None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:22.102408 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fcd970-ec02-462e-8764-58b8b6c4624a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:22.103124 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fcd970-ec02-462e-8764-58b8b6c4624a None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=f5c3ab9728f74cd8bb2614bcca9e8619) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:22.103426 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fcd970-ec02-462e-8764-58b8b6c4624a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:22.104838 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fcd970-ec02-462e-8764-58b8b6c4624a None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:22.118140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 439/864] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:48:21 2022] GET /identity/v3/role_assignments?user.id=f5c3ab9728f74cd8bb2614bcca9e8619&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 608 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:22.118140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d99bdc9-d422-4b53-b26e-d5c8ef221277 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:22.118140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d99bdc9-d422-4b53-b26e-d5c8ef221277 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:22.118140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d99bdc9-d422-4b53-b26e-d5c8ef221277 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:22.119178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:22.125744 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b916642f-5aa9-4301-ab03-8bcc45069f1d None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:22.135945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-0d99bdc9-d422-4b53-b26e-d5c8ef221277 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:22.138405 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-0d99bdc9-d422-4b53-b26e-d5c8ef221277 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:22.169645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f092cbe-4c87-4662-ab34-a302e7a52cde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:22.170429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f092cbe-4c87-4662-ab34-a302e7a52cde None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:22.170723 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f092cbe-4c87-4662-ab34-a302e7a52cde None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:22.171005 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f092cbe-4c87-4662-ab34-a302e7a52cde None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:22.171767 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f092cbe-4c87-4662-ab34-a302e7a52cde None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:22.172078 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f092cbe-4c87-4662-ab34-a302e7a52cde None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:22.173547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f092cbe-4c87-4662-ab34-a302e7a52cde None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:22.198613 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 426/865] 10.222.0.27 () {56 vars in 1149 bytes} [Wed Sep 7 03:48:22 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:22.200709 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f092cbe-4c87-4662-ab34-a302e7a52cde None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:22.211466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:22.254163 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 440/866] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:22 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:22.327217 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f092cbe-4c87-4662-ab34-a302e7a52cde None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:22.366485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb67941-781f-4b76-8024-ce4f9c4b5b9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:22.367097 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb67941-781f-4b76-8024-ce4f9c4b5b9f None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:22.367377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb67941-781f-4b76-8024-ce4f9c4b5b9f None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:22.367649 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb67941-781f-4b76-8024-ce4f9c4b5b9f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:22.395084 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:22.441618 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb67941-781f-4b76-8024-ce4f9c4b5b9f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:22.443401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb67941-781f-4b76-8024-ce4f9c4b5b9f None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:22.547851 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d99bdc9-d422-4b53-b26e-d5c8ef221277 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:22.550833 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 427/867] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:48:22 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 07 03:48:22.556198 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb67941-781f-4b76-8024-ce4f9c4b5b9f None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:22.630602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:22.657416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d99bdc9-d422-4b53-b26e-d5c8ef221277 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:22.671204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb67941-781f-4b76-8024-ce4f9c4b5b9f None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:22.695275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0754f64c-e2cc-4822-b169-6aa3e164523c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:22.695913 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0754f64c-e2cc-4822-b169-6aa3e164523c None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:22.696257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0754f64c-e2cc-4822-b169-6aa3e164523c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:22.696594 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0754f64c-e2cc-4822-b169-6aa3e164523c None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:22.697333 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0754f64c-e2cc-4822-b169-6aa3e164523c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:22.698872 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0754f64c-e2cc-4822-b169-6aa3e164523c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:22.709239 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516e556b-eb8f-4572-82bb-ccc1ffc6edfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:22.709873 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516e556b-eb8f-4572-82bb-ccc1ffc6edfb None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:22.710173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516e556b-eb8f-4572-82bb-ccc1ffc6edfb None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:22.710448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516e556b-eb8f-4572-82bb-ccc1ffc6edfb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:22.722000 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516e556b-eb8f-4572-82bb-ccc1ffc6edfb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:22.723597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516e556b-eb8f-4572-82bb-ccc1ffc6edfb None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:22.724681 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 441/868] 10.222.0.27 () {60 vars in 1141 bytes} [Wed Sep 7 03:48:22 2022] POST /identity/v3/services => generated 269 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:48:22.740911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-516e556b-eb8f-4572-82bb-ccc1ffc6edfb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:22.740911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:22.740911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:22.740911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:22.740911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:22.740911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:22.740911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:22.740911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:22.740911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:22.740911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:22.740911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:22.740911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:22.740911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:22.740911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:22.740911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:22.740911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:22.741975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:22.742911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:22.742911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:22.742911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:22.746615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 428/869] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:22.748773 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0754f64c-e2cc-4822-b169-6aa3e164523c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:22.798870 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:22.877404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0754f64c-e2cc-4822-b169-6aa3e164523c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:22.915544 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f6a05c-ac0d-4504-a1d3-0848cd52ff5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:22.916355 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f6a05c-ac0d-4504-a1d3-0848cd52ff5b None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:22.916539 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f6a05c-ac0d-4504-a1d3-0848cd52ff5b None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:22.916773 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f6a05c-ac0d-4504-a1d3-0848cd52ff5b None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:22.917445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f6a05c-ac0d-4504-a1d3-0848cd52ff5b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:22.917750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f6a05c-ac0d-4504-a1d3-0848cd52ff5b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:22.920119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f6a05c-ac0d-4504-a1d3-0848cd52ff5b None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:22.938232 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 442/870] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:22.939964 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2629d0d5-6d56-4b40-8a0c-a20fdb1f9880 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:22.940248 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2629d0d5-6d56-4b40-8a0c-a20fdb1f9880 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:22.940517 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2629d0d5-6d56-4b40-8a0c-a20fdb1f9880 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:22.942564 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 429/871] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:22.943666 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f6a05c-ac0d-4504-a1d3-0848cd52ff5b None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:22.949387 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc4f48b-6a45-4daa-9391-7c4f6811fd49 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:22.949944 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc4f48b-6a45-4daa-9391-7c4f6811fd49 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:22.950777 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc4f48b-6a45-4daa-9391-7c4f6811fd49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:22.971670 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-ebc4f48b-6a45-4daa-9391-7c4f6811fd49 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:22.972892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-ebc4f48b-6a45-4daa-9391-7c4f6811fd49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:23.057249 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 430/872] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:22 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:23.066604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:23.070651 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f6a05c-ac0d-4504-a1d3-0848cd52ff5b None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:23.108474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d99a22b-28f9-4196-98ff-ac8bbeed2778 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:23.109522 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d99a22b-28f9-4196-98ff-ac8bbeed2778 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:23.110076 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d99a22b-28f9-4196-98ff-ac8bbeed2778 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:23.110635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d99a22b-28f9-4196-98ff-ac8bbeed2778 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:23.118405 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d99a22b-28f9-4196-98ff-ac8bbeed2778 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:23.120321 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d99a22b-28f9-4196-98ff-ac8bbeed2778 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:23.134350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-2d99a22b-28f9-4196-98ff-ac8bbeed2778 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:23.134350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:23.134350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:23.134350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:23.134350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:23.134350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:23.134350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:23.134350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:23.134350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:23.134350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:23.134350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:23.134350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:23.134350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:23.134350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:23.134350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:23.135288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:23.136202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:23.136202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:23.136202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:23.136202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:23.136202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:23.136202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:23.136202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:23.136202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:23.139613 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 443/873] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:23.145167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc4f48b-6a45-4daa-9391-7c4f6811fd49 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:23.187053 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:23.250453 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc4f48b-6a45-4daa-9391-7c4f6811fd49 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:23.286235 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db0312f-4b65-47ee-b0b4-a48a77ce694b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:23.286843 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db0312f-4b65-47ee-b0b4-a48a77ce694b None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:23.287122 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db0312f-4b65-47ee-b0b4-a48a77ce694b None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:23.287394 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db0312f-4b65-47ee-b0b4-a48a77ce694b None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:23.288085 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db0312f-4b65-47ee-b0b4-a48a77ce694b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:23.288387 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db0312f-4b65-47ee-b0b4-a48a77ce694b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:23.289803 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db0312f-4b65-47ee-b0b4-a48a77ce694b None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:23.302262 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 431/874] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:23.311198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d99a22b-28f9-4196-98ff-ac8bbeed2778 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:23.332994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db0312f-4b65-47ee-b0b4-a48a77ce694b None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:23.429399 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d99a22b-28f9-4196-98ff-ac8bbeed2778 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:23.460867 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:23.466796 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bd9a71-582c-4202-9be3-41ce9af4a9b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:23.467253 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db0312f-4b65-47ee-b0b4-a48a77ce694b None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:23.467894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bd9a71-582c-4202-9be3-41ce9af4a9b7 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:23.468370 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bd9a71-582c-4202-9be3-41ce9af4a9b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:23.468866 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bd9a71-582c-4202-9be3-41ce9af4a9b7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:23.469791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bd9a71-582c-4202-9be3-41ce9af4a9b7 None admin] RBAC: Adding query filter params (role.id=a7b16bb097f14f78a7a31c04bd6d32bd, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=e7b8f511ee884dbfb4f56af78bf75984) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:23.470335 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bd9a71-582c-4202-9be3-41ce9af4a9b7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:23.471982 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bd9a71-582c-4202-9be3-41ce9af4a9b7 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:23.478633 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 444/875] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:48:23 2022] GET /identity/v3/role_assignments?user.id=e7b8f511ee884dbfb4f56af78bf75984&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=a7b16bb097f14f78a7a31c04bd6d32bd => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:23.510366 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a720cd-c265-48de-a5cb-fcc458b273f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:23.511352 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a720cd-c265-48de-a5cb-fcc458b273f9 None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:23.511833 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a720cd-c265-48de-a5cb-fcc458b273f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:23.512303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a720cd-c265-48de-a5cb-fcc458b273f9 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:23.513727 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a720cd-c265-48de-a5cb-fcc458b273f9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:23.515395 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a720cd-c265-48de-a5cb-fcc458b273f9 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:23.535117 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 432/876] 10.222.0.27 () {60 vars in 1141 bytes} [Wed Sep 7 03:48:23 2022] POST /identity/v3/services => generated 251 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 07 03:48:24.448548 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742e7879-c89d-4627-81b5-0f15365a1d42 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:24.449162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742e7879-c89d-4627-81b5-0f15365a1d42 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:24.449162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742e7879-c89d-4627-81b5-0f15365a1d42 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:24.451399 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 445/877] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:24.457479 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c240d6c-dd18-4cb9-a547-3b45267b8c10 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:24.458017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c240d6c-dd18-4cb9-a547-3b45267b8c10 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:24.458017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c240d6c-dd18-4cb9-a547-3b45267b8c10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:24.477918 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-3c240d6c-dd18-4cb9-a547-3b45267b8c10 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:24.479189 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-3c240d6c-dd18-4cb9-a547-3b45267b8c10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:24.559320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 433/878] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:24 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:24.831118 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742e7879-c89d-4627-81b5-0f15365a1d42 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:24.854990 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:24.940787 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742e7879-c89d-4627-81b5-0f15365a1d42 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:24.979374 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09849f2-7cbb-46c7-81d9-b383b8f46e47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:24.980039 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09849f2-7cbb-46c7-81d9-b383b8f46e47 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:24.980322 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09849f2-7cbb-46c7-81d9-b383b8f46e47 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:24.980596 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09849f2-7cbb-46c7-81d9-b383b8f46e47 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:24.981277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09849f2-7cbb-46c7-81d9-b383b8f46e47 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:24.982911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09849f2-7cbb-46c7-81d9-b383b8f46e47 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-d09849f2-7cbb-46c7-81d9-b383b8f46e47 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:24.992950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:24.994042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:24.994042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:24.994042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:24.994042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:24.994042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:24.994042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:24.994042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:24.994042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:24.994042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:24.994042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:24.994042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:24.994042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:24.994042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:24.994042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:24.994042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:24.994042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:24.994969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:24.994969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:24.994969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:24.994969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:24.994969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:24.994969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:24.994969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:24.994969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:24.994969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:24.994969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:24.994969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:24.994969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:24.994969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 07 03:48:24.994969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:24.994969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 446/879] 10.222.0.27 () {56 vars in 1126 bytes} [Wed Sep 7 03:48:24 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:25.000718 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c240d6c-dd18-4cb9-a547-3b45267b8c10 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:25.077312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a263b1-d527-4038-874e-d0a808e95180 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:25.077663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a263b1-d527-4038-874e-d0a808e95180 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:25.077879 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a263b1-d527-4038-874e-d0a808e95180 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:25.079805 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 447/880] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:48:25.087334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c31876-c797-48e9-865c-19f248395d26 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:25.087677 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c31876-c797-48e9-865c-19f248395d26 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:25.088222 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c31876-c797-48e9-865c-19f248395d26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:25.097223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:25.110409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-76c31876-c797-48e9-865c-19f248395d26 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:25.111795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-76c31876-c797-48e9-865c-19f248395d26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:25.114126 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c240d6c-dd18-4cb9-a547-3b45267b8c10 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:25.150908 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acba7d6-d7f3-4596-8703-8e3b48fb1dc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:25.151804 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acba7d6-d7f3-4596-8703-8e3b48fb1dc0 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:25.152236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acba7d6-d7f3-4596-8703-8e3b48fb1dc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:25.152717 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acba7d6-d7f3-4596-8703-8e3b48fb1dc0 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:25.153868 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acba7d6-d7f3-4596-8703-8e3b48fb1dc0 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:25.154419 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acba7d6-d7f3-4596-8703-8e3b48fb1dc0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:25.155980 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acba7d6-d7f3-4596-8703-8e3b48fb1dc0 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:25.161157 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 434/881] 10.222.0.27 () {56 vars in 1122 bytes} [Wed Sep 7 03:48:24 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:25.166767 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acba7d6-d7f3-4596-8703-8e3b48fb1dc0 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:25.202420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 448/882] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:25 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:48:25.273995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acba7d6-d7f3-4596-8703-8e3b48fb1dc0 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:25.310105 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00754293-fd64-4b2e-98c1-51aec7ae6e06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:25.310883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00754293-fd64-4b2e-98c1-51aec7ae6e06 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:25.311160 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00754293-fd64-4b2e-98c1-51aec7ae6e06 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:25.311428 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00754293-fd64-4b2e-98c1-51aec7ae6e06 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:25.312057 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00754293-fd64-4b2e-98c1-51aec7ae6e06 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:25.312359 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00754293-fd64-4b2e-98c1-51aec7ae6e06 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:25.313722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00754293-fd64-4b2e-98c1-51aec7ae6e06 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:25.318677 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 435/883] 10.222.0.27 () {56 vars in 1122 bytes} [Wed Sep 7 03:48:25 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:25.475534 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c31876-c797-48e9-865c-19f248395d26 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:25.492096 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:25.584609 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c31876-c797-48e9-865c-19f248395d26 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:25.620391 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68632cc-cc57-4488-97ae-0b590d1e5021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:25.621412 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68632cc-cc57-4488-97ae-0b590d1e5021 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:25.622476 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68632cc-cc57-4488-97ae-0b590d1e5021 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:25.623147 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68632cc-cc57-4488-97ae-0b590d1e5021 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:25.623941 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68632cc-cc57-4488-97ae-0b590d1e5021 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:25.625518 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68632cc-cc57-4488-97ae-0b590d1e5021 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:25.634964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-e68632cc-cc57-4488-97ae-0b590d1e5021 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 07 03:48:25.634964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:25.634964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:25.634964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:25.634964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:25.634964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:25.634964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:25.634964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:25.634964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 07 03:48:25.634964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:25.634964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:25.634964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:25.634964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:25.634964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:25.634964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:25.634964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:25.635888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:25.636780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:25.636780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:25.636780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:25.636780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:25.636780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:25.636780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:25.636780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:25.636780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:25.636780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:25.636780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:25.636780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:25.636780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:25.636780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 07 03:48:25.636780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:25.639836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 449/884] 10.222.0.27 () {56 vars in 1132 bytes} [Wed Sep 7 03:48:25 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:25.650181 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00754293-fd64-4b2e-98c1-51aec7ae6e06 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:25.664574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:25.759706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00754293-fd64-4b2e-98c1-51aec7ae6e06 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:25.796290 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5cc0e4-4296-4803-a18e-91645e75c3a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:25.796902 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5cc0e4-4296-4803-a18e-91645e75c3a4 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:25.797178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5cc0e4-4296-4803-a18e-91645e75c3a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:25.797451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5cc0e4-4296-4803-a18e-91645e75c3a4 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:25.798085 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5cc0e4-4296-4803-a18e-91645e75c3a4 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:25.798421 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5cc0e4-4296-4803-a18e-91645e75c3a4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:25.799778 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5cc0e4-4296-4803-a18e-91645e75c3a4 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:25.805018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 436/885] 10.222.0.27 () {56 vars in 1126 bytes} [Wed Sep 7 03:48:25 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:25.810747 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68632cc-cc57-4488-97ae-0b590d1e5021 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:25.815294 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8161d4-ee5f-477a-850f-48f60da14f91 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:25.815580 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8161d4-ee5f-477a-850f-48f60da14f91 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:25.815854 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8161d4-ee5f-477a-850f-48f60da14f91 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:25.817410 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 437/886] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:25.823245 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13880620-628d-4d44-90ea-7304c075bd84 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:25.823529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13880620-628d-4d44-90ea-7304c075bd84 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:25.823803 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13880620-628d-4d44-90ea-7304c075bd84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:25.843325 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-13880620-628d-4d44-90ea-7304c075bd84 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:25.844577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-13880620-628d-4d44-90ea-7304c075bd84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:25.901704 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:25.917371 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 438/887] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:25 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:48:25.919008 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2738978-c896-4ed2-b77c-bd47d50bf00d None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:25.919527 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2738978-c896-4ed2-b77c-bd47d50bf00d None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:25.920009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2738978-c896-4ed2-b77c-bd47d50bf00d None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:25.921718 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 439/888] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:25.928506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab283fcd-287b-4449-bb31-e77d5394816c None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:25.929030 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab283fcd-287b-4449-bb31-e77d5394816c None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:25.929753 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab283fcd-287b-4449-bb31-e77d5394816c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:25.933231 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68632cc-cc57-4488-97ae-0b590d1e5021 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:25.950952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:25.953431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-ab283fcd-287b-4449-bb31-e77d5394816c None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:25.955183 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-ab283fcd-287b-4449-bb31-e77d5394816c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:25.970083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea875c1-6dba-4788-a0d2-236c270a91c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:25.970810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea875c1-6dba-4788-a0d2-236c270a91c6 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:25.971165 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea875c1-6dba-4788-a0d2-236c270a91c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:25.971513 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea875c1-6dba-4788-a0d2-236c270a91c6 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:25.972220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea875c1-6dba-4788-a0d2-236c270a91c6 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:25.972612 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea875c1-6dba-4788-a0d2-236c270a91c6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:25.974055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea875c1-6dba-4788-a0d2-236c270a91c6 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:25.984600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 450/889] 10.222.0.27 () {56 vars in 1126 bytes} [Wed Sep 7 03:48:25 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:25.985986 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea875c1-6dba-4788-a0d2-236c270a91c6 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:26.054650 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 440/890] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:25 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:48:26.102297 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea875c1-6dba-4788-a0d2-236c270a91c6 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:26.137907 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625bb477-a348-4e21-8c12-c364d1cc026c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:26.138587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625bb477-a348-4e21-8c12-c364d1cc026c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:26.138931 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625bb477-a348-4e21-8c12-c364d1cc026c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:26.139209 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625bb477-a348-4e21-8c12-c364d1cc026c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:26.139970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625bb477-a348-4e21-8c12-c364d1cc026c None admin] RBAC: Adding query filter params (interface=public, service_id=d366c68605c846d0b92aed6134d8c81c, region_id=RegionOne) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:26.140334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625bb477-a348-4e21-8c12-c364d1cc026c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:26.141717 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625bb477-a348-4e21-8c12-c364d1cc026c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:26.147348 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 451/891] 10.222.0.27 () {56 vars in 1251 bytes} [Wed Sep 7 03:48:25 2022] GET /identity/v3/endpoints?service_id=d366c68605c846d0b92aed6134d8c81c&interface=public®ion_id=RegionOne => generated 194 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:26.155225 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab283fcd-287b-4449-bb31-e77d5394816c None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:26.200196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625bb477-a348-4e21-8c12-c364d1cc026c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:26.209919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:26.270106 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab283fcd-287b-4449-bb31-e77d5394816c None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:26.305857 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdd200a-4a8c-411d-8223-cdaf4a82da10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:26.306802 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdd200a-4a8c-411d-8223-cdaf4a82da10 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:26.307280 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdd200a-4a8c-411d-8223-cdaf4a82da10 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:26.307748 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdd200a-4a8c-411d-8223-cdaf4a82da10 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:26.308544 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdd200a-4a8c-411d-8223-cdaf4a82da10 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:26.310836 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdd200a-4a8c-411d-8223-cdaf4a82da10 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:26.312839 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:26.318481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 441/892] 10.222.0.27 () {56 vars in 1087 bytes} [Wed Sep 7 03:48:26 2022] GET /identity/v3/services => generated 1268 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 07 03:48:26.334158 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625bb477-a348-4e21-8c12-c364d1cc026c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:26.361164 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdd200a-4a8c-411d-8223-cdaf4a82da10 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:26.368110 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497b6356-4818-40d8-8bed-c4948ffe675a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:26.368763 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497b6356-4818-40d8-8bed-c4948ffe675a None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:26.369047 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497b6356-4818-40d8-8bed-c4948ffe675a None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:26.369324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497b6356-4818-40d8-8bed-c4948ffe675a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:26.382407 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497b6356-4818-40d8-8bed-c4948ffe675a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:26.388326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-497b6356-4818-40d8-8bed-c4948ffe675a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 07 03:48:26.388326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:26.388326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:26.388326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:26.388326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:26.388326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:26.388326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:26.388326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:26.388326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:26.388326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:26.388326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:26.388326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:26.388326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:26.388326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:26.388326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:26.388326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:26.389353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:26.389353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:26.389353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:26.389353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:26.389353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:26.389353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:26.389353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:26.389353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:26.389353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:26.389353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:26.389353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:26.389353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:26.389353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:26.389353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:26.389353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:26.389353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:26.395490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:26.395490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:26.395490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:26.395490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:26.395490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:26.395490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:26.395490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:26.395490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:26.395490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:26.395490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:26.395490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 07 03:48:26.395490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:26.395490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 452/893] 10.222.0.27 () {56 vars in 1120 bytes} [Wed Sep 7 03:48:26 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:26.395490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497b6356-4818-40d8-8bed-c4948ffe675a None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:26.490678 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:26.511655 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdd200a-4a8c-411d-8223-cdaf4a82da10 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:26.543839 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497b6356-4818-40d8-8bed-c4948ffe675a None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:26.554874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c8bfec-31f3-46f5-894f-19c0a327467e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:26.555611 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c8bfec-31f3-46f5-894f-19c0a327467e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:26.556703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c8bfec-31f3-46f5-894f-19c0a327467e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:26.556703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c8bfec-31f3-46f5-894f-19c0a327467e None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:26.556967 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c8bfec-31f3-46f5-894f-19c0a327467e None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:26.558515 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c8bfec-31f3-46f5-894f-19c0a327467e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-51c8bfec-31f3-46f5-894f-19c0a327467e None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:26.575887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:26.576996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:26.576996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:26.576996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:26.576996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:26.576996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:26.576996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:26.576996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:26.576996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:26.576996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:26.576996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:26.576996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:26.576996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:26.576996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:26.576996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:26.576996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:26.576996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:26.577832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:26.577832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:26.577832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:26.577832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:26.577832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:26.577832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:26.577832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:26.577832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:26.577832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:26.577832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:26.577832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:26.577832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:26.577832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 07 03:48:26.577832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:26.577832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 442/894] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:48:26 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:26.583817 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c8bfec-31f3-46f5-894f-19c0a327467e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:26.593300 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57088aea-4323-4fa2-b03a-f06582695506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:26.594279 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57088aea-4323-4fa2-b03a-f06582695506 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:26.594767 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57088aea-4323-4fa2-b03a-f06582695506 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:26.595746 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57088aea-4323-4fa2-b03a-f06582695506 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:26.596618 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57088aea-4323-4fa2-b03a-f06582695506 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:26.597117 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57088aea-4323-4fa2-b03a-f06582695506 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:26.599548 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57088aea-4323-4fa2-b03a-f06582695506 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:26.611807 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 453/895] 10.222.0.27 () {56 vars in 1115 bytes} [Wed Sep 7 03:48:26 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:26.617232 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57088aea-4323-4fa2-b03a-f06582695506 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:26.662527 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:26.718318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c8bfec-31f3-46f5-894f-19c0a327467e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:26.752108 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57088aea-4323-4fa2-b03a-f06582695506 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:26.760339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f725c1-b8ec-4ec3-b86e-607b0a7d18d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:26.760955 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f725c1-b8ec-4ec3-b86e-607b0a7d18d5 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:26.761235 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f725c1-b8ec-4ec3-b86e-607b0a7d18d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:26.761513 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f725c1-b8ec-4ec3-b86e-607b0a7d18d5 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:26.762185 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f725c1-b8ec-4ec3-b86e-607b0a7d18d5 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:26.763422 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f725c1-b8ec-4ec3-b86e-607b0a7d18d5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:26.764978 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f725c1-b8ec-4ec3-b86e-607b0a7d18d5 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:26.771758 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 443/896] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:48:26 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:26.776432 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f725c1-b8ec-4ec3-b86e-607b0a7d18d5 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:26.786568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:26.798658 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a2437e-b24b-462c-975e-3d6874451785 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:26.799893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a2437e-b24b-462c-975e-3d6874451785 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:26.800770 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a2437e-b24b-462c-975e-3d6874451785 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:26.801258 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a2437e-b24b-462c-975e-3d6874451785 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:26.816559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a2437e-b24b-462c-975e-3d6874451785 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:26.819003 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a2437e-b24b-462c-975e-3d6874451785 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:26.839680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-22a2437e-b24b-462c-975e-3d6874451785 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:26.839680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:26.839680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:26.839680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:26.839680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:26.839680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:26.839680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:26.839680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:26.839680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:26.839680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:26.839680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:26.839680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:26.839680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:26.839680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:26.839680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:26.839680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:26.841586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:26.842513 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:26.842513 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:26.842513 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:26.846754 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 454/897] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:26.848735 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a2437e-b24b-462c-975e-3d6874451785 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:26.907610 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f725c1-b8ec-4ec3-b86e-607b0a7d18d5 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:26.941813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:26.952705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626973bb-8ad0-4b75-b257-73c4b3a604c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:26.953332 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626973bb-8ad0-4b75-b257-73c4b3a604c1 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:26.953616 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626973bb-8ad0-4b75-b257-73c4b3a604c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:26.954186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626973bb-8ad0-4b75-b257-73c4b3a604c1 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:26.954844 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626973bb-8ad0-4b75-b257-73c4b3a604c1 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:26.955143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626973bb-8ad0-4b75-b257-73c4b3a604c1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:26.956907 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626973bb-8ad0-4b75-b257-73c4b3a604c1 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:26.963743 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 444/898] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:48:26 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:26.969599 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626973bb-8ad0-4b75-b257-73c4b3a604c1 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:26.989959 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a2437e-b24b-462c-975e-3d6874451785 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:27.033994 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c442ab-6e05-4d12-a0ae-0c742d251374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:27.034649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c442ab-6e05-4d12-a0ae-0c742d251374 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:27.034930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c442ab-6e05-4d12-a0ae-0c742d251374 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:27.035206 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c442ab-6e05-4d12-a0ae-0c742d251374 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:27.035857 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c442ab-6e05-4d12-a0ae-0c742d251374 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:27.036160 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c442ab-6e05-4d12-a0ae-0c742d251374 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:27.037541 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c442ab-6e05-4d12-a0ae-0c742d251374 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:27.057978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 455/899] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:27.060383 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c442ab-6e05-4d12-a0ae-0c742d251374 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:27.094475 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626973bb-8ad0-4b75-b257-73c4b3a604c1 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:27.109680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:27.133436 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0264d6c-474e-4bc9-a391-298b55501fe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:27.134317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0264d6c-474e-4bc9-a391-298b55501fe0 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:27.134783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0264d6c-474e-4bc9-a391-298b55501fe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:27.135205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0264d6c-474e-4bc9-a391-298b55501fe0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:27.136058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0264d6c-474e-4bc9-a391-298b55501fe0 None admin] RBAC: Adding query filter params (interface=public, service_id=a3786bdcef7346079ef6087207fca765, region_id=RegionOne) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:27.136516 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0264d6c-474e-4bc9-a391-298b55501fe0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:27.138501 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0264d6c-474e-4bc9-a391-298b55501fe0 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:27.146869 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 445/900] 10.222.0.27 () {56 vars in 1251 bytes} [Wed Sep 7 03:48:26 2022] GET /identity/v3/endpoints?service_id=a3786bdcef7346079ef6087207fca765&interface=public®ion_id=RegionOne => generated 194 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:27.154892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0264d6c-474e-4bc9-a391-298b55501fe0 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:27.184351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c442ab-6e05-4d12-a0ae-0c742d251374 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:27.229296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:27.233256 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e211d888-fcfa-4d78-9309-9d75d01395ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:27.234010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e211d888-fcfa-4d78-9309-9d75d01395ba None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:27.234418 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e211d888-fcfa-4d78-9309-9d75d01395ba None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:27.234947 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e211d888-fcfa-4d78-9309-9d75d01395ba None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:27.244735 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e211d888-fcfa-4d78-9309-9d75d01395ba None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:27.246669 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e211d888-fcfa-4d78-9309-9d75d01395ba None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:27.262151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-e211d888-fcfa-4d78-9309-9d75d01395ba None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 07 03:48:27.262151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:27.262151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:27.262151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:27.262151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:27.262151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:27.262151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:27.262151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:27.262151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:27.262151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:27.262151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:27.262151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:27.262151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:27.262151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:27.262151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:27.262151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:27.264339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:27.266137 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:27.266137 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:27.266137 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:27.266137 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:27.266137 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:27.266137 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:27.266137 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:27.266137 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:27.266137 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 07 03:48:27.266137 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:27.271975 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 456/901] 10.222.0.27 () {56 vars in 1117 bytes} [Wed Sep 7 03:48:27 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:27.273449 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e211d888-fcfa-4d78-9309-9d75d01395ba None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:27.292984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0264d6c-474e-4bc9-a391-298b55501fe0 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:27.334316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04a127c-129f-448e-97a1-196e74292851 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:27.335476 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04a127c-129f-448e-97a1-196e74292851 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:27.335945 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04a127c-129f-448e-97a1-196e74292851 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:27.336682 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04a127c-129f-448e-97a1-196e74292851 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:27.337932 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04a127c-129f-448e-97a1-196e74292851 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:27.340109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04a127c-129f-448e-97a1-196e74292851 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:27.353988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 446/902] 10.222.0.27 () {56 vars in 1087 bytes} [Wed Sep 7 03:48:27 2022] GET /identity/v3/services => generated 1268 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 07 03:48:27.401057 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:27.405165 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e211d888-fcfa-4d78-9309-9d75d01395ba None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:27.454570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbedfac6-f1cb-4a1d-90db-95bdf8fae320 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:27.455242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbedfac6-f1cb-4a1d-90db-95bdf8fae320 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:27.455520 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbedfac6-f1cb-4a1d-90db-95bdf8fae320 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:27.455793 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbedfac6-f1cb-4a1d-90db-95bdf8fae320 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:27.456553 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbedfac6-f1cb-4a1d-90db-95bdf8fae320 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:27.456862 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbedfac6-f1cb-4a1d-90db-95bdf8fae320 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:27.458473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbedfac6-f1cb-4a1d-90db-95bdf8fae320 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:27.480786 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 457/903] 10.222.0.27 () {56 vars in 1149 bytes} [Wed Sep 7 03:48:27 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:27.487329 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04a127c-129f-448e-97a1-196e74292851 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:27.553224 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081d4621-9f5e-401b-b7d1-59ec5a9f4e72 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:27.553764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081d4621-9f5e-401b-b7d1-59ec5a9f4e72 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:27.553764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081d4621-9f5e-401b-b7d1-59ec5a9f4e72 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:27.555386 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 458/904] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:48:27.561524 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de73772-f3ad-4b6c-903b-8f6050ade5af None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:27.562085 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de73772-f3ad-4b6c-903b-8f6050ade5af None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:27.562622 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de73772-f3ad-4b6c-903b-8f6050ade5af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:27.585471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-4de73772-f3ad-4b6c-903b-8f6050ade5af None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:27.586686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-4de73772-f3ad-4b6c-903b-8f6050ade5af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:27.611990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04a127c-129f-448e-97a1-196e74292851 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:27.635093 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:27.650046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110b3e6e-8e03-408c-a273-53fdfc9f920d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:27.650686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110b3e6e-8e03-408c-a273-53fdfc9f920d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:27.650965 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110b3e6e-8e03-408c-a273-53fdfc9f920d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:27.651239 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110b3e6e-8e03-408c-a273-53fdfc9f920d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:27.690574 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 459/905] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:27 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:48:27.725493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110b3e6e-8e03-408c-a273-53fdfc9f920d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:27.727070 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110b3e6e-8e03-408c-a273-53fdfc9f920d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:27.832994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 447/906] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:48:27 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:48:27.838255 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de73772-f3ad-4b6c-903b-8f6050ade5af None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:27.939239 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:27.958150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de73772-f3ad-4b6c-903b-8f6050ade5af None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:27.974473 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110b3e6e-8e03-408c-a273-53fdfc9f920d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:27.996324 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774f8798-dfa3-41d8-8b6a-02a4ade3b923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:27.997704 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774f8798-dfa3-41d8-8b6a-02a4ade3b923 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:27.998690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:27.999346 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774f8798-dfa3-41d8-8b6a-02a4ade3b923 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:27.999523 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774f8798-dfa3-41d8-8b6a-02a4ade3b923 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:28.013619 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774f8798-dfa3-41d8-8b6a-02a4ade3b923 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:28.015128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774f8798-dfa3-41d8-8b6a-02a4ade3b923 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:28.033743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-774f8798-dfa3-41d8-8b6a-02a4ade3b923 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:28.033743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:28.033743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:28.033743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:28.033743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:28.033743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:28.033743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:28.033743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:28.033743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:28.033743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:28.033743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:28.033743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:28.033743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:28.033743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:28.033743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:28.033743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:28.034755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:28.035661 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:28.035661 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:28.035661 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:28.035831 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 460/907] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:28.041018 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774f8798-dfa3-41d8-8b6a-02a4ade3b923 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:28.091554 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110b3e6e-8e03-408c-a273-53fdfc9f920d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:28.129038 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57102feb-a085-4533-8725-3407acb4aa2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:28.129653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57102feb-a085-4533-8725-3407acb4aa2a None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:28.129936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57102feb-a085-4533-8725-3407acb4aa2a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:28.130328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57102feb-a085-4533-8725-3407acb4aa2a None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:28.131021 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57102feb-a085-4533-8725-3407acb4aa2a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:28.132388 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57102feb-a085-4533-8725-3407acb4aa2a None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:28.150889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 448/908] 10.222.0.27 () {60 vars in 1141 bytes} [Wed Sep 7 03:48:27 2022] POST /identity/v3/services => generated 241 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 07 03:48:28.186516 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774f8798-dfa3-41d8-8b6a-02a4ade3b923 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:28.234980 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d571b284-a6fd-48c4-915c-0d743191771d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:28.235529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d571b284-a6fd-48c4-915c-0d743191771d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:28.235808 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d571b284-a6fd-48c4-915c-0d743191771d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:28.236083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d571b284-a6fd-48c4-915c-0d743191771d None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:28.236739 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d571b284-a6fd-48c4-915c-0d743191771d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:28.237044 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d571b284-a6fd-48c4-915c-0d743191771d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:28.238488 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d571b284-a6fd-48c4-915c-0d743191771d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:28.247515 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:28.253588 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 461/909] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:28.259456 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57102feb-a085-4533-8725-3407acb4aa2a None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:28.356496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:28.379976 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57102feb-a085-4533-8725-3407acb4aa2a None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:28.415874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd64ca3-b7af-4515-96dd-1f2414743b60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:28.416510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd64ca3-b7af-4515-96dd-1f2414743b60 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:28.416788 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd64ca3-b7af-4515-96dd-1f2414743b60 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:28.417062 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd64ca3-b7af-4515-96dd-1f2414743b60 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:28.424144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd64ca3-b7af-4515-96dd-1f2414743b60 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:28.425573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd64ca3-b7af-4515-96dd-1f2414743b60 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:28.437511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-bbd64ca3-b7af-4515-96dd-1f2414743b60 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:28.437511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:28.437511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:28.437511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:28.437511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:28.437511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:28.437511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:28.437511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:28.437511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:28.437511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:28.437511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:28.437511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:28.437511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:28.437511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:28.437511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:28.438454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:28.439424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:28.439424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:28.439424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:28.439424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:28.439424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:28.439424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:28.439424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:28.439424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:28.439424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 449/910] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:28.444722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d571b284-a6fd-48c4-915c-0d743191771d None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:28.551977 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d571b284-a6fd-48c4-915c-0d743191771d None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:28.591204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66988ea0-ee8e-457c-bca8-cc4de2f89832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:28.591878 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66988ea0-ee8e-457c-bca8-cc4de2f89832 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:28.592167 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66988ea0-ee8e-457c-bca8-cc4de2f89832 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:28.592446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66988ea0-ee8e-457c-bca8-cc4de2f89832 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:28.593146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66988ea0-ee8e-457c-bca8-cc4de2f89832 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:28.593454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66988ea0-ee8e-457c-bca8-cc4de2f89832 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:28.594914 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66988ea0-ee8e-457c-bca8-cc4de2f89832 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:28.611718 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 462/911] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:28.617395 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd64ca3-b7af-4515-96dd-1f2414743b60 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:28.664987 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611d8f18-7689-426e-8440-aea1927dda5d None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:28.665298 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611d8f18-7689-426e-8440-aea1927dda5d None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:28.665549 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611d8f18-7689-426e-8440-aea1927dda5d None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:28.667285 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 463/912] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:28.672623 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c578c252-542f-4f10-81ee-cc5eb2abf497 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:28.672910 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c578c252-542f-4f10-81ee-cc5eb2abf497 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:28.673192 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c578c252-542f-4f10-81ee-cc5eb2abf497 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:28.694945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-c578c252-542f-4f10-81ee-cc5eb2abf497 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:28.696561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-c578c252-542f-4f10-81ee-cc5eb2abf497 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:28.732332 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd64ca3-b7af-4515-96dd-1f2414743b60 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:28.770929 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:28.775559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375d5648-f2f1-4d31-ad31-532138e02a26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:28.776729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375d5648-f2f1-4d31-ad31-532138e02a26 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:28.777233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375d5648-f2f1-4d31-ad31-532138e02a26 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:28.777884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375d5648-f2f1-4d31-ad31-532138e02a26 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:28.779355 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375d5648-f2f1-4d31-ad31-532138e02a26 None admin] RBAC: Adding query filter params (role.id=d2787a643a3148c882b7a96a59e0a5e3, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=e7b8f511ee884dbfb4f56af78bf75984) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:28.780092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375d5648-f2f1-4d31-ad31-532138e02a26 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:28.782197 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375d5648-f2f1-4d31-ad31-532138e02a26 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:28.789713 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 450/913] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:48:28 2022] GET /identity/v3/role_assignments?user.id=e7b8f511ee884dbfb4f56af78bf75984&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=d2787a643a3148c882b7a96a59e0a5e3 => generated 259 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:28.798171 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 464/914] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:28 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:48:29.063507 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375d5648-f2f1-4d31-ad31-532138e02a26 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:29.068910 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:29.172907 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375d5648-f2f1-4d31-ad31-532138e02a26 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:29.208715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fc45f5-ac48-4f22-8588-931f4fef0fcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:29.209411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fc45f5-ac48-4f22-8588-931f4fef0fcd None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:29.209691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fc45f5-ac48-4f22-8588-931f4fef0fcd None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:29.210022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fc45f5-ac48-4f22-8588-931f4fef0fcd None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:29.210647 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fc45f5-ac48-4f22-8588-931f4fef0fcd None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:29.212011 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fc45f5-ac48-4f22-8588-931f4fef0fcd None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:29.221730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-50fc45f5-ac48-4f22-8588-931f4fef0fcd None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 07 03:48:29.221730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:29.221730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:29.221730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:29.221730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:29.221730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:29.221730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:29.221730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:29.221730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 07 03:48:29.221730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:29.221730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:29.221730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:29.221730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:29.221730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:29.221730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:29.221730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:29.222703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:29.223582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:29.223582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:29.223582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:29.223582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:29.223582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:29.223582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:29.223582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:29.223582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:29.223582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:29.223582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:29.223582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:29.223582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:29.223582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 07 03:48:29.223582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:29.223582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 451/915] 10.222.0.27 () {56 vars in 1132 bytes} [Wed Sep 7 03:48:29 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:29.230496 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c578c252-542f-4f10-81ee-cc5eb2abf497 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:29.349015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c578c252-542f-4f10-81ee-cc5eb2abf497 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:29.384776 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07aab62-a0b7-404a-bc94-06efc1963b02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:29.385638 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07aab62-a0b7-404a-bc94-06efc1963b02 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:29.386062 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07aab62-a0b7-404a-bc94-06efc1963b02 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:29.386504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07aab62-a0b7-404a-bc94-06efc1963b02 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:29.387352 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07aab62-a0b7-404a-bc94-06efc1963b02 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:29.387815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07aab62-a0b7-404a-bc94-06efc1963b02 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:29.389334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07aab62-a0b7-404a-bc94-06efc1963b02 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:29.394649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 465/916] 10.222.0.27 () {56 vars in 1126 bytes} [Wed Sep 7 03:48:29 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:29.404271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fc45f5-ac48-4f22-8588-931f4fef0fcd None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:29.476785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:29.513118 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fc45f5-ac48-4f22-8588-931f4fef0fcd None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:29.549448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68cf9aa-e4b2-482a-97c4-89ca435463a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:29.550061 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68cf9aa-e4b2-482a-97c4-89ca435463a1 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:29.550375 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68cf9aa-e4b2-482a-97c4-89ca435463a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:29.550702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68cf9aa-e4b2-482a-97c4-89ca435463a1 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:29.551492 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68cf9aa-e4b2-482a-97c4-89ca435463a1 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:29.551850 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68cf9aa-e4b2-482a-97c4-89ca435463a1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:29.553287 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68cf9aa-e4b2-482a-97c4-89ca435463a1 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:29.559299 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 452/917] 10.222.0.27 () {56 vars in 1126 bytes} [Wed Sep 7 03:48:29 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:29.565309 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07aab62-a0b7-404a-bc94-06efc1963b02 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:29.619351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:29.679887 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07aab62-a0b7-404a-bc94-06efc1963b02 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:29.722146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d6d192-38d1-48cc-8931-91d2e42f1d1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:29.723131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d6d192-38d1-48cc-8931-91d2e42f1d1c None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:29.723614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d6d192-38d1-48cc-8931-91d2e42f1d1c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:29.724122 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d6d192-38d1-48cc-8931-91d2e42f1d1c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:29.724993 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d6d192-38d1-48cc-8931-91d2e42f1d1c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:29.726593 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d6d192-38d1-48cc-8931-91d2e42f1d1c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:29.748945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 466/918] 10.222.0.27 () {60 vars in 1144 bytes} [Wed Sep 7 03:48:29 2022] POST /identity/v3/endpoints => generated 347 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:48:29.784999 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e848782f-518b-454c-bd2a-1e4b82b086d8 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:29.785549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e848782f-518b-454c-bd2a-1e4b82b086d8 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:29.785549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e848782f-518b-454c-bd2a-1e4b82b086d8 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:29.787200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 453/919] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:29.793253 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8119c73a-1350-49f7-8e00-b2823a70b402 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:29.794472 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8119c73a-1350-49f7-8e00-b2823a70b402 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:29.795036 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8119c73a-1350-49f7-8e00-b2823a70b402 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:29.816350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-8119c73a-1350-49f7-8e00-b2823a70b402 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:29.817998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-8119c73a-1350-49f7-8e00-b2823a70b402 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:29.909249 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 467/920] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:29 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:30.177744 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e848782f-518b-454c-bd2a-1e4b82b086d8 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:30.286936 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e848782f-518b-454c-bd2a-1e4b82b086d8 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:30.316338 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:30.321623 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbea41af-09fa-4e48-8766-eff0213c2c0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:30.322283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbea41af-09fa-4e48-8766-eff0213c2c0b None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:30.322793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbea41af-09fa-4e48-8766-eff0213c2c0b None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:30.323023 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbea41af-09fa-4e48-8766-eff0213c2c0b None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:30.323952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbea41af-09fa-4e48-8766-eff0213c2c0b None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:30.325955 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbea41af-09fa-4e48-8766-eff0213c2c0b None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-dbea41af-09fa-4e48-8766-eff0213c2c0b None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:30.335994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:30.337050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:30.337050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:30.337050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:30.337050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:30.337050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:30.337050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:30.337050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:30.337050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:30.337050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:30.337050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:30.337050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:30.337050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:30.337050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:30.337050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:30.337050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:30.337050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:30.337892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:30.337892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:30.337892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:30.337892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:30.337892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:30.337892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:30.337892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:30.337892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:30.337892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:30.337892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:30.337892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:30.337892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:30.337892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 07 03:48:30.337892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:30.337892 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 454/921] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:48:30 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:30.343935 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8119c73a-1350-49f7-8e00-b2823a70b402 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:30.381164 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:30.457441 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8119c73a-1350-49f7-8e00-b2823a70b402 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:30.467020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20361cf4-a684-4196-9c06-da0543ffbda8 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:30.467304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20361cf4-a684-4196-9c06-da0543ffbda8 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:30.467583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20361cf4-a684-4196-9c06-da0543ffbda8 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:30.469268 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 455/922] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:30.474459 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f8bc41-248b-46cf-8504-f5bf552fcd6e None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:30.474740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f8bc41-248b-46cf-8504-f5bf552fcd6e None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:30.475120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f8bc41-248b-46cf-8504-f5bf552fcd6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:30.494829 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-32f8bc41-248b-46cf-8504-f5bf552fcd6e None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:30.495944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ae584f-d6ee-455d-a075-aa9276fc4166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:30.496171 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-32f8bc41-248b-46cf-8504-f5bf552fcd6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:30.496653 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ae584f-d6ee-455d-a075-aa9276fc4166 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:30.497141 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ae584f-d6ee-455d-a075-aa9276fc4166 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:30.497479 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ae584f-d6ee-455d-a075-aa9276fc4166 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:30.498143 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ae584f-d6ee-455d-a075-aa9276fc4166 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:30.498463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ae584f-d6ee-455d-a075-aa9276fc4166 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:30.499851 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ae584f-d6ee-455d-a075-aa9276fc4166 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:30.509566 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 468/923] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:48:30 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:30.510504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ae584f-d6ee-455d-a075-aa9276fc4166 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:30.584462 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 456/924] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:30 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:48:30.620555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ae584f-d6ee-455d-a075-aa9276fc4166 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:30.632512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:30.657632 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0304280-c630-43fd-a589-3a4c6e4c451b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:30.658276 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0304280-c630-43fd-a589-3a4c6e4c451b None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:30.658557 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0304280-c630-43fd-a589-3a4c6e4c451b None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:30.658829 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0304280-c630-43fd-a589-3a4c6e4c451b None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:30.659564 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0304280-c630-43fd-a589-3a4c6e4c451b None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:30.659871 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0304280-c630-43fd-a589-3a4c6e4c451b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:30.661227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0304280-c630-43fd-a589-3a4c6e4c451b None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:30.666384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 469/925] 10.222.0.27 () {56 vars in 1114 bytes} [Wed Sep 7 03:48:30 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:30.675206 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f8bc41-248b-46cf-8504-f5bf552fcd6e None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:30.782319 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f8bc41-248b-46cf-8504-f5bf552fcd6e None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:30.807618 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:30.817445 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6185905-5c96-4484-a875-e5dcef05bb9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:30.818052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6185905-5c96-4484-a875-e5dcef05bb9a None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:30.818354 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6185905-5c96-4484-a875-e5dcef05bb9a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:30.818626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6185905-5c96-4484-a875-e5dcef05bb9a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:30.819349 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6185905-5c96-4484-a875-e5dcef05bb9a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:30.820704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6185905-5c96-4484-a875-e5dcef05bb9a None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:30.843625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 457/926] 10.222.0.27 () {60 vars in 1144 bytes} [Wed Sep 7 03:48:30 2022] POST /identity/v3/endpoints => generated 346 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:48:30.895646 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0304280-c630-43fd-a589-3a4c6e4c451b None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:31.015586 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56330fd5-5195-4f92-b2c2-4525e08412f9 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:31.015951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56330fd5-5195-4f92-b2c2-4525e08412f9 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:31.016115 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56330fd5-5195-4f92-b2c2-4525e08412f9 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:31.017624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0304280-c630-43fd-a589-3a4c6e4c451b None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:31.017812 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 458/927] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:48:31.023693 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d615172-0d37-453f-9275-2f72de246803 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:31.024103 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d615172-0d37-453f-9275-2f72de246803 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:31.024495 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d615172-0d37-453f-9275-2f72de246803 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:31.046385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-8d615172-0d37-453f-9275-2f72de246803 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:31.047536 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-8d615172-0d37-453f-9275-2f72de246803 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:31.069459 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa7bf4b-6170-4ea4-8915-a47d6d399693 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:31.070398 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa7bf4b-6170-4ea4-8915-a47d6d399693 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:31.070832 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa7bf4b-6170-4ea4-8915-a47d6d399693 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:31.071235 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa7bf4b-6170-4ea4-8915-a47d6d399693 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:31.071983 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa7bf4b-6170-4ea4-8915-a47d6d399693 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:31.073551 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa7bf4b-6170-4ea4-8915-a47d6d399693 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-bfa7bf4b-6170-4ea4-8915-a47d6d399693 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:31.087665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:31.088705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:31.088705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:31.088705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:31.088705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:31.088705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:31.088705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:31.088705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:31.088705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:31.088705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:31.088705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:31.088705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:31.088705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:31.088705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:31.088705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:31.088705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:31.088705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:31.089529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:31.089529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:31.089529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:31.089529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:31.089529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:31.089529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:31.089529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:31.089529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:31.089529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:31.089529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:31.089529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:31.089529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:31.089529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 07 03:48:31.089529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:31.089529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 470/928] 10.222.0.27 () {56 vars in 1126 bytes} [Wed Sep 7 03:48:30 2022] GET /identity/v3/services/object-store => generated 93 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:31.095346 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa7bf4b-6170-4ea4-8915-a47d6d399693 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:31.125055 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 459/929] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:31 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:31.125685 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:31.210258 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa7bf4b-6170-4ea4-8915-a47d6d399693 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:31.246041 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a56048d-fac4-4c1c-b8f2-23af012eeb6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:31.247097 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a56048d-fac4-4c1c-b8f2-23af012eeb6d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:31.247583 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a56048d-fac4-4c1c-b8f2-23af012eeb6d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:31.248047 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a56048d-fac4-4c1c-b8f2-23af012eeb6d None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:31.248874 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a56048d-fac4-4c1c-b8f2-23af012eeb6d None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:31.249360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a56048d-fac4-4c1c-b8f2-23af012eeb6d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:31.250922 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a56048d-fac4-4c1c-b8f2-23af012eeb6d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:31.256545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 471/930] 10.222.0.27 () {56 vars in 1122 bytes} [Wed Sep 7 03:48:31 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:31.266137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d615172-0d37-453f-9275-2f72de246803 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:31.368615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d615172-0d37-453f-9275-2f72de246803 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:31.383143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:31.406643 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d45f746-102f-4c0e-b140-d54afffd3f10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:31.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d45f746-102f-4c0e-b140-d54afffd3f10 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:31.407794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d45f746-102f-4c0e-b140-d54afffd3f10 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:31.408072 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d45f746-102f-4c0e-b140-d54afffd3f10 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:31.408719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d45f746-102f-4c0e-b140-d54afffd3f10 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:31.409023 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d45f746-102f-4c0e-b140-d54afffd3f10 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:31.410673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d45f746-102f-4c0e-b140-d54afffd3f10 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:31.411777 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a56048d-fac4-4c1c-b8f2-23af012eeb6d None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:31.422154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 460/931] 10.222.0.27 () {56 vars in 1122 bytes} [Wed Sep 7 03:48:31 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:31.425595 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d45f746-102f-4c0e-b140-d54afffd3f10 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:31.527219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:31.545834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d45f746-102f-4c0e-b140-d54afffd3f10 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:31.553220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a56048d-fac4-4c1c-b8f2-23af012eeb6d None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:31.584142 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ce5e54-a9fa-4bd5-a9e7-272dbb180e9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:31.584853 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ce5e54-a9fa-4bd5-a9e7-272dbb180e9e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:31.585135 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ce5e54-a9fa-4bd5-a9e7-272dbb180e9e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:31.585406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ce5e54-a9fa-4bd5-a9e7-272dbb180e9e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:31.586106 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ce5e54-a9fa-4bd5-a9e7-272dbb180e9e None admin] RBAC: Adding query filter params (interface=public, service_id=25aa18d1c2c84ceea0d1e1baa89b7e83, region_id=RegionOne) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:31.586449 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ce5e54-a9fa-4bd5-a9e7-272dbb180e9e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:31.587869 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ce5e54-a9fa-4bd5-a9e7-272dbb180e9e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:31.597789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 461/932] 10.222.0.27 () {56 vars in 1251 bytes} [Wed Sep 7 03:48:31 2022] GET /identity/v3/endpoints?service_id=25aa18d1c2c84ceea0d1e1baa89b7e83&interface=public®ion_id=RegionOne => generated 194 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:31.600027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ce5e54-a9fa-4bd5-a9e7-272dbb180e9e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:31.602015 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91829424-3807-4665-abe8-e534500d5c5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:31.603257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91829424-3807-4665-abe8-e534500d5c5c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:31.604716 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91829424-3807-4665-abe8-e534500d5c5c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:31.605218 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91829424-3807-4665-abe8-e534500d5c5c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:31.620180 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91829424-3807-4665-abe8-e534500d5c5c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:31.628568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-91829424-3807-4665-abe8-e534500d5c5c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 07 03:48:31.628568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:31.628568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:31.628568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:31.628568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:31.628568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:31.628568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:31.628568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:31.628568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:31.628568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:31.628568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:31.628568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:31.628568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:31.628568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:31.628568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:31.628568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:31.629945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:31.629945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:31.629945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:31.629945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:31.629945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:31.629945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:31.629945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:31.629945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:31.629945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:31.629945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:31.629945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:31.629945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:31.629945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:31.629945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:31.629945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:31.629945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:31.630828 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:31.630828 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:31.630828 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:31.630828 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:31.630828 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:31.630828 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:31.630828 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:31.630828 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:31.630828 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:31.630828 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:31.630828 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 07 03:48:31.630828 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:31.634190 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 472/933] 10.222.0.27 () {56 vars in 1120 bytes} [Wed Sep 7 03:48:31 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:31.637647 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91829424-3807-4665-abe8-e534500d5c5c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:31.652565 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:31.730436 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ce5e54-a9fa-4bd5-a9e7-272dbb180e9e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:31.756772 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91829424-3807-4665-abe8-e534500d5c5c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:31.777549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bbd706-1aa8-4fe3-813b-5b594ad394c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:31.778645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bbd706-1aa8-4fe3-813b-5b594ad394c0 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:31.779730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bbd706-1aa8-4fe3-813b-5b594ad394c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:31.780230 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bbd706-1aa8-4fe3-813b-5b594ad394c0 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:31.781298 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1bbd706-1aa8-4fe3-813b-5b594ad394c0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:31.783949 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1bbd706-1aa8-4fe3-813b-5b594ad394c0 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:31.793280 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 462/934] 10.222.0.27 () {56 vars in 1087 bytes} [Wed Sep 7 03:48:31 2022] GET /identity/v3/services => generated 1497 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 07 03:48:31.804742 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385d6294-d204-42fb-aec5-cf44cbbaa0ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:31.804742 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385d6294-d204-42fb-aec5-cf44cbbaa0ad None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:31.804742 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385d6294-d204-42fb-aec5-cf44cbbaa0ad None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:31.804742 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385d6294-d204-42fb-aec5-cf44cbbaa0ad None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:31.805192 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385d6294-d204-42fb-aec5-cf44cbbaa0ad None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:31.805192 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385d6294-d204-42fb-aec5-cf44cbbaa0ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:31.805192 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385d6294-d204-42fb-aec5-cf44cbbaa0ad None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:31.814391 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 473/935] 10.222.0.27 () {56 vars in 1115 bytes} [Wed Sep 7 03:48:31 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:31.820193 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bbd706-1aa8-4fe3-813b-5b594ad394c0 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:31.944641 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bbd706-1aa8-4fe3-813b-5b594ad394c0 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:31.947599 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:31.989427 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1caa7e-bdac-4468-86eb-b551ff5b4a8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:31.989427 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1caa7e-bdac-4468-86eb-b551ff5b4a8e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:31.989427 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1caa7e-bdac-4468-86eb-b551ff5b4a8e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:31.989427 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1caa7e-bdac-4468-86eb-b551ff5b4a8e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:32.005399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1caa7e-bdac-4468-86eb-b551ff5b4a8e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:32.007441 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1caa7e-bdac-4468-86eb-b551ff5b4a8e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:32.024168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-ce1caa7e-bdac-4468-86eb-b551ff5b4a8e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:32.024168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:32.024168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:32.024168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:32.024168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:32.024168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:32.024168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:32.024168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:32.024168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:32.024168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:32.024168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:32.024168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:32.024168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:32.024168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:32.024168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:32.024168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:32.025367 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:32.026496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:32.026496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:32.026496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:32.028431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 463/936] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:32.034438 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385d6294-d204-42fb-aec5-cf44cbbaa0ad None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:32.046170 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:32.050511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e9e007-dee9-4263-9446-4ece66bf901e None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:32.050804 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e9e007-dee9-4263-9446-4ece66bf901e None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:32.051195 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e9e007-dee9-4263-9446-4ece66bf901e None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:32.052882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 464/937] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:32.058492 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff54be3-54c1-425b-96ed-6c716f33ec39 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:32.060024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff54be3-54c1-425b-96ed-6c716f33ec39 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:32.060792 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff54be3-54c1-425b-96ed-6c716f33ec39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:32.081451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-fff54be3-54c1-425b-96ed-6c716f33ec39 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:32.083107 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-fff54be3-54c1-425b-96ed-6c716f33ec39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:32.164540 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 465/938] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:32 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:32.169418 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385d6294-d204-42fb-aec5-cf44cbbaa0ad None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:32.208540 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc9fdd0-fd2e-401c-9e66-3198b1d0e17d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:32.209508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc9fdd0-fd2e-401c-9e66-3198b1d0e17d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:32.209987 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc9fdd0-fd2e-401c-9e66-3198b1d0e17d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:32.210665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc9fdd0-fd2e-401c-9e66-3198b1d0e17d None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:32.211514 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc9fdd0-fd2e-401c-9e66-3198b1d0e17d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:32.212040 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc9fdd0-fd2e-401c-9e66-3198b1d0e17d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:32.213750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc9fdd0-fd2e-401c-9e66-3198b1d0e17d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:32.228756 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 474/939] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:32.236630 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff54be3-54c1-425b-96ed-6c716f33ec39 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:32.331894 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:32.348825 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff54be3-54c1-425b-96ed-6c716f33ec39 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:32.387009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfd3b62-5470-47ef-bc65-30a2cc6adcad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:32.389044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfd3b62-5470-47ef-bc65-30a2cc6adcad None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:32.391021 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfd3b62-5470-47ef-bc65-30a2cc6adcad None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:32.391021 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfd3b62-5470-47ef-bc65-30a2cc6adcad None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:32.398147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfd3b62-5470-47ef-bc65-30a2cc6adcad None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:32.400952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfd3b62-5470-47ef-bc65-30a2cc6adcad None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:32.414384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-5bfd3b62-5470-47ef-bc65-30a2cc6adcad None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 07 03:48:32.414384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:32.414384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:32.414384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:32.414384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:32.414384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:32.414384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:32.414384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:32.414384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:32.414384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:32.414384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:32.414384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:32.414384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:32.414384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:32.414384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:32.414384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:32.415695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:32.417783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:32.417783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:32.417783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:32.417783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:32.417783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:32.417783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:32.417783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:32.417783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:32.417783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 07 03:48:32.417783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:32.417783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 466/940] 10.222.0.27 () {56 vars in 1117 bytes} [Wed Sep 7 03:48:32 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:32.421205 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc9fdd0-fd2e-401c-9e66-3198b1d0e17d None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:32.491009 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:32.527079 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc9fdd0-fd2e-401c-9e66-3198b1d0e17d None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:32.541898 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfd3b62-5470-47ef-bc65-30a2cc6adcad None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:32.569008 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999855eb-a642-4ed2-90c9-b23b37e0b9f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:32.570087 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999855eb-a642-4ed2-90c9-b23b37e0b9f6 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:32.570760 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999855eb-a642-4ed2-90c9-b23b37e0b9f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:32.571240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999855eb-a642-4ed2-90c9-b23b37e0b9f6 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:32.572185 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999855eb-a642-4ed2-90c9-b23b37e0b9f6 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:32.572762 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999855eb-a642-4ed2-90c9-b23b37e0b9f6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:32.575659 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999855eb-a642-4ed2-90c9-b23b37e0b9f6 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:32.599699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 475/941] 10.222.0.27 () {56 vars in 1149 bytes} [Wed Sep 7 03:48:32 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:32.604975 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999855eb-a642-4ed2-90c9-b23b37e0b9f6 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:32.663481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfd3b62-5470-47ef-bc65-30a2cc6adcad None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:32.701763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a0be9c-a894-4b3d-8f06-10bab7c4138a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:32.702461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a0be9c-a894-4b3d-8f06-10bab7c4138a None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:32.702741 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a0be9c-a894-4b3d-8f06-10bab7c4138a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:32.703017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a0be9c-a894-4b3d-8f06-10bab7c4138a None admin] PATH_INFO: `/v3/services/compute` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:32.703605 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a0be9c-a894-4b3d-8f06-10bab7c4138a None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:32.705266 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a0be9c-a894-4b3d-8f06-10bab7c4138a None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-30a0be9c-a894-4b3d-8f06-10bab7c4138a None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:32.720399 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:32.721446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:32.721446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:32.721446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:32.721446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:32.721446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:32.721446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:32.721446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:32.721446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:32.721446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:32.721446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:32.721446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:32.721446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:32.721446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:32.721446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:32.721446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:32.721446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:32.722324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:32.722324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:32.722324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:32.722324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:32.722324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:32.722324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:32.722324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:32.722324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:32.722324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:32.722324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:32.722324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:32.722324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:32.722324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 07 03:48:32.722324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:32.725149 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 467/942] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:32 2022] GET /identity/v3/services/compute => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:32.731705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a0be9c-a894-4b3d-8f06-10bab7c4138a None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:32.743573 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999855eb-a642-4ed2-90c9-b23b37e0b9f6 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:32.748964 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:32.770507 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:32.784717 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17dcefcc-3b98-4954-9b43-c20c12dd8a4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:32.785368 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17dcefcc-3b98-4954-9b43-c20c12dd8a4e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:32.785678 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17dcefcc-3b98-4954-9b43-c20c12dd8a4e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:32.785946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17dcefcc-3b98-4954-9b43-c20c12dd8a4e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:32.800645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17dcefcc-3b98-4954-9b43-c20c12dd8a4e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:32.802721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17dcefcc-3b98-4954-9b43-c20c12dd8a4e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:32.825906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-17dcefcc-3b98-4954-9b43-c20c12dd8a4e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:32.825906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:32.825906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:32.825906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:32.825906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:32.825906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:32.825906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:32.825906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:32.825906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:32.825906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:32.825906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:32.825906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:32.825906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:32.825906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:32.825906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:32.825906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:32.827474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:32.829246 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:32.829246 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:32.829246 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:32.832288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 476/943] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:32.833218 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17dcefcc-3b98-4954-9b43-c20c12dd8a4e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:32.864688 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a0be9c-a894-4b3d-8f06-10bab7c4138a None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:32.906687 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8eb28d-725d-448e-a14d-7f9e5e1ccd5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:32.907682 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8eb28d-725d-448e-a14d-7f9e5e1ccd5b None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:32.908164 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8eb28d-725d-448e-a14d-7f9e5e1ccd5b None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:32.908635 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8eb28d-725d-448e-a14d-7f9e5e1ccd5b None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:32.909459 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8eb28d-725d-448e-a14d-7f9e5e1ccd5b None admin] RBAC: Adding query filter params (name=compute) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:32.909980 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8eb28d-725d-448e-a14d-7f9e5e1ccd5b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:32.911541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8eb28d-725d-448e-a14d-7f9e5e1ccd5b None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:32.921284 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 468/944] 10.222.0.27 () {56 vars in 1112 bytes} [Wed Sep 7 03:48:32 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:32.922847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8eb28d-725d-448e-a14d-7f9e5e1ccd5b None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:32.959588 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17dcefcc-3b98-4954-9b43-c20c12dd8a4e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:32.997942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670a54c3-e168-4130-9fc7-e2dcf53ea238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:32.998957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670a54c3-e168-4130-9fc7-e2dcf53ea238 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:32.999440 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670a54c3-e168-4130-9fc7-e2dcf53ea238 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:33.000016 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670a54c3-e168-4130-9fc7-e2dcf53ea238 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:33.000959 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670a54c3-e168-4130-9fc7-e2dcf53ea238 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:33.001535 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670a54c3-e168-4130-9fc7-e2dcf53ea238 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:33.003238 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670a54c3-e168-4130-9fc7-e2dcf53ea238 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:33.022309 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 477/945] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:33.025835 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670a54c3-e168-4130-9fc7-e2dcf53ea238 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:33.028897 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:33.067124 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:33.068624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8eb28d-725d-448e-a14d-7f9e5e1ccd5b None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:33.113516 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60137bbb-d163-48fb-be74-55c429e17573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:33.114142 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60137bbb-d163-48fb-be74-55c429e17573 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:33.114491 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60137bbb-d163-48fb-be74-55c429e17573 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:33.114770 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60137bbb-d163-48fb-be74-55c429e17573 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:33.115515 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60137bbb-d163-48fb-be74-55c429e17573 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:33.115823 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60137bbb-d163-48fb-be74-55c429e17573 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:33.117179 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60137bbb-d163-48fb-be74-55c429e17573 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:33.134306 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 469/946] 10.222.0.27 () {56 vars in 1112 bytes} [Wed Sep 7 03:48:32 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:33.145870 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670a54c3-e168-4130-9fc7-e2dcf53ea238 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:33.154904 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb01f42e-7609-4d85-9202-be3014cb8edc None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:33.155242 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb01f42e-7609-4d85-9202-be3014cb8edc None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:33.155448 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb01f42e-7609-4d85-9202-be3014cb8edc None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:33.161388 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 470/947] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:33 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:48:33.163381 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3449e0a-9242-4b34-832d-4d6b59d5e74c None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:33.163852 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3449e0a-9242-4b34-832d-4d6b59d5e74c None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:33.164319 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3449e0a-9242-4b34-832d-4d6b59d5e74c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:33.185763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-f3449e0a-9242-4b34-832d-4d6b59d5e74c None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:33.187047 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-f3449e0a-9242-4b34-832d-4d6b59d5e74c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:33.194698 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59ea919-35a1-49b5-b9f2-33856c7a37ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:33.195781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59ea919-35a1-49b5-b9f2-33856c7a37ba None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:33.197025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59ea919-35a1-49b5-b9f2-33856c7a37ba None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:33.197526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59ea919-35a1-49b5-b9f2-33856c7a37ba None admin] PATH_INFO: `/v3/projects/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:33.208927 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59ea919-35a1-49b5-b9f2-33856c7a37ba None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:33.210865 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59ea919-35a1-49b5-b9f2-33856c7a37ba None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:33.228465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-a59ea919-35a1-49b5-b9f2-33856c7a37ba None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:33.228465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:33.228465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:33.228465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:33.228465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:33.228465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:33.228465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:33.228465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:33.228465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:33.228465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:33.228465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:33.228465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:33.228465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:33.228465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:33.228465 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:33.229420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:33.230362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:33.230362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:33.230362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:33.230362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:33.230362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:33.230362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:33.230362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:33.230362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:33.237491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 478/948] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:33.237765 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59ea919-35a1-49b5-b9f2-33856c7a37ba None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:33.280223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 471/949] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:33 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:33.356178 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:33.362743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59ea919-35a1-49b5-b9f2-33856c7a37ba None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:33.399392 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e3373a-6544-42e4-86ce-d1b784cf426f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:33.400009 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e3373a-6544-42e4-86ce-d1b784cf426f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:33.400288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e3373a-6544-42e4-86ce-d1b784cf426f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:33.400615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e3373a-6544-42e4-86ce-d1b784cf426f None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:33.401313 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e3373a-6544-42e4-86ce-d1b784cf426f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:33.401721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e3373a-6544-42e4-86ce-d1b784cf426f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:33.403894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e3373a-6544-42e4-86ce-d1b784cf426f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:33.416872 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 479/950] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:33.423123 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3449e0a-9242-4b34-832d-4d6b59d5e74c None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:33.498558 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:33.535976 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3449e0a-9242-4b34-832d-4d6b59d5e74c None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:33.557213 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e3373a-6544-42e4-86ce-d1b784cf426f None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:33.575782 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efab5ac5-29ec-4f8a-be15-7a512e803ece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:33.576741 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efab5ac5-29ec-4f8a-be15-7a512e803ece None admin] REQUEST_METHOD: `PUT` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:33.577361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efab5ac5-29ec-4f8a-be15-7a512e803ece None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:33.577850 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efab5ac5-29ec-4f8a-be15-7a512e803ece None admin] PATH_INFO: `/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/e7b8f511ee884dbfb4f56af78bf75984/roles/d2787a643a3148c882b7a96a59e0a5e3` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:33.581357 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efab5ac5-29ec-4f8a-be15-7a512e803ece None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4816e95144485e8135e5b8f68c3662, user_id=e7b8f511ee884dbfb4f56af78bf75984, role_id=d2787a643a3148c882b7a96a59e0a5e3)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:33.583394 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efab5ac5-29ec-4f8a-be15-7a512e803ece None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:33.603758 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 472/951] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:48:33 2022] PUT /identity/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/e7b8f511ee884dbfb4f56af78bf75984/roles/d2787a643a3148c882b7a96a59e0a5e3 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:48:33.691728 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e3373a-6544-42e4-86ce-d1b784cf426f None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:33.732300 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4649d1-6053-47e1-b4d7-c95dfb206df9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:33.732935 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4649d1-6053-47e1-b4d7-c95dfb206df9 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:33.733213 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4649d1-6053-47e1-b4d7-c95dfb206df9 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:33.733486 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4649d1-6053-47e1-b4d7-c95dfb206df9 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:33.734242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4649d1-6053-47e1-b4d7-c95dfb206df9 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:33.735736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4649d1-6053-47e1-b4d7-c95dfb206df9 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:33.738586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: WARNING keystone.server.flask.common [None req-8f4649d1-6053-47e1-b4d7-c95dfb206df9 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 07 03:48:33.760722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.sql.core [None req-8f4649d1-6053-47e1-b4d7-c95dfb206df9 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 07 03:48:33.760722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 07 03:48:33.760722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [parameters: {'id': 'da374198928d445c91b7002a29a02c43', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 07 03:48:33.760722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81391) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 07 03:48:33.771385 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-8f4649d1-6053-47e1-b4d7-c95dfb206df9 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id da374198928d445c91b7002a29a02c43.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id da374198928d445c91b7002a29a02c43. Sep 07 03:48:33.771385 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:33.771385 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 07 03:48:33.771385 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 07 03:48:33.771385 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 07 03:48:33.771385 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 07 03:48:33.771385 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 07 03:48:33.771385 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application result = self._query(query) Sep 07 03:48:33.771385 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 07 03:48:33.771385 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application conn.query(q) Sep 07 03:48:33.771385 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 07 03:48:33.771385 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 07 03:48:33.771385 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application result.read() Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 07 03:48:33.772257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application next(self.gen) Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application yield resource Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application next(self.gen) Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self.session.rollback() Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self.force_reraise() Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise self.value Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application session.commit() Sep 07 03:48:33.773194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._prepare_impl() Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self.session.flush() Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._flush(objects) Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application compat.raise_( Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application flush_context.execute() Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 07 03:48:33.774932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rec.execute(self) Sep 07 03:48:33.777109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 07 03:48:33.777109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 07 03:48:33.777109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 07 03:48:33.777109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 07 03:48:33.777109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 07 03:48:33.777109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 07 03:48:33.777109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 07 03:48:33.777109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 07 03:48:33.777109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 07 03:48:33.777109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 07 03:48:33.777109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 07 03:48:33.777109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 07 03:48:33.777109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 07 03:48:33.777109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 07 03:48:33.777109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application result = self._query(query) Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application conn.query(q) Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application result.read() Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 07 03:48:33.778894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 07 03:48:33.780526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 07 03:48:33.780526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 07 03:48:33.780526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 07 03:48:33.780526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 07 03:48:33.780526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 07 03:48:33.780526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 07 03:48:33.780526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 07 03:48:33.780526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 07 03:48:33.780526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [parameters: {'id': 'da374198928d445c91b7002a29a02c43', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 07 03:48:33.780526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 07 03:48:33.780526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:33.780526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:33.780526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:33.780526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:33.780526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:33.782645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:33.783566 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:33.783566 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 07 03:48:33.783566 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 07 03:48:33.783566 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:33.783566 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:33.783566 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 07 03:48:33.783566 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 07 03:48:33.783566 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id da374198928d445c91b7002a29a02c43. Sep 07 03:48:33.783566 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:33.783566 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 480/952] 10.222.0.27 () {60 vars in 1140 bytes} [Wed Sep 7 03:48:33 2022] POST /identity/v3/projects => generated 262 bytes in 216 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:33.784104 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efab5ac5-29ec-4f8a-be15-7a512e803ece None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:33.899534 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efab5ac5-29ec-4f8a-be15-7a512e803ece None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:33.929234 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:33.940178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d26cdd-57b4-48c9-9369-309f74feb53f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:33.941113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d26cdd-57b4-48c9-9369-309f74feb53f None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:33.941610 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d26cdd-57b4-48c9-9369-309f74feb53f None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:33.942095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d26cdd-57b4-48c9-9369-309f74feb53f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:33.949772 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d26cdd-57b4-48c9-9369-309f74feb53f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:33.951802 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d26cdd-57b4-48c9-9369-309f74feb53f None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:33.964089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-23d26cdd-57b4-48c9-9369-309f74feb53f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:33.964089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:33.964089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:33.964089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:33.964089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:33.964089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:33.964089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:33.964089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:33.964089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:33.964089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:33.964089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:33.964089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:33.964089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:33.964089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:33.964089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:33.965045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:33.966579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:33.966579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:33.966579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:33.966579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:33.966579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:33.966579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:33.966579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:33.966579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:33.969129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 473/953] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:33.975218 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4649d1-6053-47e1-b4d7-c95dfb206df9 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:33.991790 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:34.084896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4649d1-6053-47e1-b4d7-c95dfb206df9 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:34.122433 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c05887a-cb2f-4696-a1a8-7a990752aa0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:34.123491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c05887a-cb2f-4696-a1a8-7a990752aa0f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:34.123974 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c05887a-cb2f-4696-a1a8-7a990752aa0f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:34.124476 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c05887a-cb2f-4696-a1a8-7a990752aa0f None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:34.125397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c05887a-cb2f-4696-a1a8-7a990752aa0f None admin] RBAC: Adding query filter params (name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:34.125965 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c05887a-cb2f-4696-a1a8-7a990752aa0f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:34.127990 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c05887a-cb2f-4696-a1a8-7a990752aa0f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:34.152858 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 481/954] 10.222.0.27 () {56 vars in 1112 bytes} [Wed Sep 7 03:48:33 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:34.264945 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704d2f04-0125-4542-8dc1-c1a29bbfddf6 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:34.265510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704d2f04-0125-4542-8dc1-c1a29bbfddf6 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:34.265972 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704d2f04-0125-4542-8dc1-c1a29bbfddf6 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:34.267809 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 474/955] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:34.276080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c3bf06-bb65-4079-b466-7c2c23493961 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:34.276708 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c3bf06-bb65-4079-b466-7c2c23493961 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:34.277205 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c3bf06-bb65-4079-b466-7c2c23493961 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:34.298412 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-80c3bf06-bb65-4079-b466-7c2c23493961 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:34.300433 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-80c3bf06-bb65-4079-b466-7c2c23493961 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:34.373051 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 482/956] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:34 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:34.648895 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704d2f04-0125-4542-8dc1-c1a29bbfddf6 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:34.757709 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704d2f04-0125-4542-8dc1-c1a29bbfddf6 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:34.793174 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06295db1-7f0e-48ba-bea8-bff3c0951819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:34.794724 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06295db1-7f0e-48ba-bea8-bff3c0951819 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:34.795364 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06295db1-7f0e-48ba-bea8-bff3c0951819 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:34.795835 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06295db1-7f0e-48ba-bea8-bff3c0951819 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:34.796679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06295db1-7f0e-48ba-bea8-bff3c0951819 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:34.798389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06295db1-7f0e-48ba-bea8-bff3c0951819 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-06295db1-7f0e-48ba-bea8-bff3c0951819 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:34.808052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:34.809377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:34.809377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:34.809377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:34.809377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:34.809377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:34.809377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:34.809377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:34.809377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:34.809377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:34.809377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:34.809377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:34.809377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:34.809377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:34.809377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:34.809377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:34.809377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:34.810236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:34.810236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:34.810236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:34.810236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:34.810236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:34.810236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:34.810236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:34.810236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:34.810236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:34.810236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:34.810236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:34.810236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:34.810236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 07 03:48:34.810236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:34.812925 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 475/957] 10.222.0.27 () {56 vars in 1126 bytes} [Wed Sep 7 03:48:34 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:34.819077 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c3bf06-bb65-4079-b466-7c2c23493961 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:34.851533 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:34.925105 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c3bf06-bb65-4079-b466-7c2c23493961 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:34.961221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4784ad7-8008-4cec-b208-1189c516c8ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:34.962349 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4784ad7-8008-4cec-b208-1189c516c8ca None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:34.962831 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4784ad7-8008-4cec-b208-1189c516c8ca None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:34.963300 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4784ad7-8008-4cec-b208-1189c516c8ca None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:34.964151 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4784ad7-8008-4cec-b208-1189c516c8ca None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:34.964660 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4784ad7-8008-4cec-b208-1189c516c8ca None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:34.966344 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4784ad7-8008-4cec-b208-1189c516c8ca None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:34.972048 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 483/958] 10.222.0.27 () {56 vars in 1122 bytes} [Wed Sep 7 03:48:34 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:34.977893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06295db1-7f0e-48ba-bea8-bff3c0951819 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:35.035844 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:35.085950 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06295db1-7f0e-48ba-bea8-bff3c0951819 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:35.120181 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6225c1f-21c6-4899-a630-18b0a83f2ba8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:35.121222 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6225c1f-21c6-4899-a630-18b0a83f2ba8 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:35.122479 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6225c1f-21c6-4899-a630-18b0a83f2ba8 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:35.122988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6225c1f-21c6-4899-a630-18b0a83f2ba8 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:35.123869 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6225c1f-21c6-4899-a630-18b0a83f2ba8 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:35.124381 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6225c1f-21c6-4899-a630-18b0a83f2ba8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:35.126020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6225c1f-21c6-4899-a630-18b0a83f2ba8 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:35.132440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 476/959] 10.222.0.27 () {56 vars in 1122 bytes} [Wed Sep 7 03:48:34 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:35.138907 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4784ad7-8008-4cec-b208-1189c516c8ca None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:35.206775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:35.242317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4784ad7-8008-4cec-b208-1189c516c8ca None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:35.284231 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a20a0f-46ed-40ef-a4f1-16df312bda03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:35.285077 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a20a0f-46ed-40ef-a4f1-16df312bda03 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:35.285501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a20a0f-46ed-40ef-a4f1-16df312bda03 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:35.285917 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a20a0f-46ed-40ef-a4f1-16df312bda03 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:35.286784 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a20a0f-46ed-40ef-a4f1-16df312bda03 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:35.288302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a20a0f-46ed-40ef-a4f1-16df312bda03 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:35.311849 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 484/960] 10.222.0.27 () {60 vars in 1144 bytes} [Wed Sep 7 03:48:35 2022] POST /identity/v3/endpoints => generated 349 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:48:35.470957 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a371ee4-fdba-4587-bdc7-391ad4a681ca None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:35.471144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a371ee4-fdba-4587-bdc7-391ad4a681ca None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:35.471458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a371ee4-fdba-4587-bdc7-391ad4a681ca None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:35.473138 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 477/961] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:35.478950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e83dd8-1dc0-4865-840a-8f3b15e1b4bb None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:35.479568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e83dd8-1dc0-4865-840a-8f3b15e1b4bb None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:35.480059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e83dd8-1dc0-4865-840a-8f3b15e1b4bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:35.501069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-42e83dd8-1dc0-4865-840a-8f3b15e1b4bb None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:35.502654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-42e83dd8-1dc0-4865-840a-8f3b15e1b4bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:35.585724 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 485/962] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:35 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:48:35.858218 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a371ee4-fdba-4587-bdc7-391ad4a681ca None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:35.895103 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd151bae-3879-4b2a-8104-f6da01d40523 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:35.895634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd151bae-3879-4b2a-8104-f6da01d40523 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:35.895634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd151bae-3879-4b2a-8104-f6da01d40523 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:35.897506 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 486/963] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:35.904063 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639fa7d4-c478-4cbd-af1f-91ff9a47c3cf None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:35.904339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639fa7d4-c478-4cbd-af1f-91ff9a47c3cf None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:35.904610 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639fa7d4-c478-4cbd-af1f-91ff9a47c3cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:35.930109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-639fa7d4-c478-4cbd-af1f-91ff9a47c3cf None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:35.931572 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-639fa7d4-c478-4cbd-af1f-91ff9a47c3cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:35.977051 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a371ee4-fdba-4587-bdc7-391ad4a681ca None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:35.988659 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:35.999798 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:36.024131 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7517703-d3e5-4b56-bfbb-1cdaf284971a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:36.025119 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7517703-d3e5-4b56-bfbb-1cdaf284971a None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:36.025609 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7517703-d3e5-4b56-bfbb-1cdaf284971a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:36.026185 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7517703-d3e5-4b56-bfbb-1cdaf284971a None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:36.027126 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7517703-d3e5-4b56-bfbb-1cdaf284971a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:36.029553 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7517703-d3e5-4b56-bfbb-1cdaf284971a None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:36.030214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 487/964] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:35 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:48:36.047351 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 478/965] 10.222.0.27 () {60 vars in 1141 bytes} [Wed Sep 7 03:48:35 2022] POST /identity/v3/services => generated 242 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:48:36.305326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639fa7d4-c478-4cbd-af1f-91ff9a47c3cf None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:36.415297 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639fa7d4-c478-4cbd-af1f-91ff9a47c3cf None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:36.447339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e4484f-94d1-4522-a17f-849de881f8c8 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:36.447668 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e4484f-94d1-4522-a17f-849de881f8c8 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:36.447893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e4484f-94d1-4522-a17f-849de881f8c8 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:36.454189 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 479/966] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:36.459740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e231ffe9-2720-4393-94ba-45122632f14d None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:36.460027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e231ffe9-2720-4393-94ba-45122632f14d None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:36.460306 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e231ffe9-2720-4393-94ba-45122632f14d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:36.463986 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2349c2b9-3602-437f-b3ad-e18758d80e3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:36.464876 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2349c2b9-3602-437f-b3ad-e18758d80e3c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:36.465933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2349c2b9-3602-437f-b3ad-e18758d80e3c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:36.466617 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2349c2b9-3602-437f-b3ad-e18758d80e3c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:36.479339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2349c2b9-3602-437f-b3ad-e18758d80e3c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:36.480542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-e231ffe9-2720-4393-94ba-45122632f14d None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:36.481781 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-e231ffe9-2720-4393-94ba-45122632f14d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:36.486424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-2349c2b9-3602-437f-b3ad-e18758d80e3c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 07 03:48:36.486424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:36.486424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:36.486424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:36.486424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:36.486424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:36.486424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:36.486424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:36.486424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:36.486424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:36.486424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:36.486424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:36.486424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:36.486424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:36.486424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:36.486424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:36.487356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:36.487356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:36.487356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:36.487356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:36.487356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:36.487356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:36.487356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:36.487356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:36.487356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:36.487356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:36.487356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:36.487356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:36.487356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:36.487356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:36.487356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:36.487356 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:36.488204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:36.488204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:36.488204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:36.488204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:36.488204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:36.488204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:36.488204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:36.488204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:36.488204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:36.488204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:36.488204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 07 03:48:36.488204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:36.491238 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 488/967] 10.222.0.27 () {56 vars in 1120 bytes} [Wed Sep 7 03:48:36 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:36.498034 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2349c2b9-3602-437f-b3ad-e18758d80e3c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:36.543549 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:36.563734 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 480/968] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:36 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:36.620139 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2349c2b9-3602-437f-b3ad-e18758d80e3c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:36.657017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6248d202-7934-4692-beea-be32a8fa0296 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:36.657644 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6248d202-7934-4692-beea-be32a8fa0296 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:36.657932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6248d202-7934-4692-beea-be32a8fa0296 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:36.658236 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6248d202-7934-4692-beea-be32a8fa0296 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:36.658916 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6248d202-7934-4692-beea-be32a8fa0296 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:36.659226 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6248d202-7934-4692-beea-be32a8fa0296 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:36.660610 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6248d202-7934-4692-beea-be32a8fa0296 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:36.669938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 489/969] 10.222.0.27 () {56 vars in 1115 bytes} [Wed Sep 7 03:48:36 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:36.675325 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e231ffe9-2720-4393-94ba-45122632f14d None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:36.781622 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e231ffe9-2720-4393-94ba-45122632f14d None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:36.791541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:36.822813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97fcda7-0738-4a5a-8830-6530946c4bda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:36.823359 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97fcda7-0738-4a5a-8830-6530946c4bda None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:36.823635 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97fcda7-0738-4a5a-8830-6530946c4bda None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:36.823905 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97fcda7-0738-4a5a-8830-6530946c4bda None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:36.834715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97fcda7-0738-4a5a-8830-6530946c4bda None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:36.836159 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97fcda7-0738-4a5a-8830-6530946c4bda None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:36.843414 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6248d202-7934-4692-beea-be32a8fa0296 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:36.855010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-a97fcda7-0738-4a5a-8830-6530946c4bda None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:36.855010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:36.855010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:36.855010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:36.855010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:36.855010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:36.855010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:36.855010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:36.855010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:36.855010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:36.855010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:36.855010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:36.855010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:36.855010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:36.855010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:36.855010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:36.856082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:36.857087 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:36.857087 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:36.857087 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:36.857087 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 481/970] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:36.862609 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97fcda7-0738-4a5a-8830-6530946c4bda None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:36.951726 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:36.958873 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6248d202-7934-4692-beea-be32a8fa0296 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:36.996753 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0247354-3a1e-40b9-96ce-4a2102566dda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:36.997386 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97fcda7-0738-4a5a-8830-6530946c4bda None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:36.998089 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0247354-3a1e-40b9-96ce-4a2102566dda None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:36.998652 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0247354-3a1e-40b9-96ce-4a2102566dda None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:36.999129 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0247354-3a1e-40b9-96ce-4a2102566dda None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:36.999996 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0247354-3a1e-40b9-96ce-4a2102566dda None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:37.001521 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0247354-3a1e-40b9-96ce-4a2102566dda None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:37.017538 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.sql.core [None req-c0247354-3a1e-40b9-96ce-4a2102566dda None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 07 03:48:37.017538 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 07 03:48:37.017538 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [parameters: {'user_id': 'c813db9fc796412483ce9c5c4ec49628', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 07 03:48:37.017538 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81391) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 07 03:48:37.025740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-c0247354-3a1e-40b9-96ce-4a2102566dda None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 07 03:48:37.025740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:37.025740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 07 03:48:37.025740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 07 03:48:37.025740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 07 03:48:37.025740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 07 03:48:37.025740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 07 03:48:37.025740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application result = self._query(query) Sep 07 03:48:37.025740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 07 03:48:37.025740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application conn.query(q) Sep 07 03:48:37.025740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 07 03:48:37.025740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 07 03:48:37.025740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 07 03:48:37.025740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application result.read() Sep 07 03:48:37.025740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 07 03:48:37.026712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application next(self.gen) Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application yield resource Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application next(self.gen) Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self.session.rollback() Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self.force_reraise() Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise self.value Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application session.commit() Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 07 03:48:37.027846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._prepare_impl() Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self.session.flush() Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._flush(objects) Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application compat.raise_( Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application flush_context.execute() Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rec.execute(self) Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 07 03:48:37.028744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 07 03:48:37.029626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 07 03:48:37.029626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 07 03:48:37.029626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 07 03:48:37.029626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 07 03:48:37.029626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 07 03:48:37.029626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 07 03:48:37.029626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 07 03:48:37.029626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 07 03:48:37.029626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 07 03:48:37.029626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 07 03:48:37.029626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 07 03:48:37.029626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 07 03:48:37.029626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 07 03:48:37.029626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 07 03:48:37.029626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 07 03:48:37.029626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception Sep 07 03:48:37.030517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 07 03:48:37.030517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 07 03:48:37.030517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 07 03:48:37.030517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 07 03:48:37.030517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 07 03:48:37.030517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application result = self._query(query) Sep 07 03:48:37.030517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 07 03:48:37.030517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application conn.query(q) Sep 07 03:48:37.030517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 07 03:48:37.030517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 07 03:48:37.030517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 07 03:48:37.030517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application result.read() Sep 07 03:48:37.030517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 07 03:48:37.030517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 07 03:48:37.030517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 07 03:48:37.030517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 07 03:48:37.031351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 07 03:48:37.031351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 07 03:48:37.031351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 07 03:48:37.031351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 07 03:48:37.031351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 07 03:48:37.031351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 07 03:48:37.031351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [parameters: {'user_id': 'c813db9fc796412483ce9c5c4ec49628', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 07 03:48:37.031351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 07 03:48:37.031351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:37.031351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:37.031351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:37.031351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:37.031351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:37.031351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:37.031351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 07 03:48:37.032159 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 07 03:48:37.033077 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 07 03:48:37.033077 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 07 03:48:37.033077 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 07 03:48:37.033077 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 07 03:48:37.033077 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 07 03:48:37.033077 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:37.040055 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80beda7-9c53-4a23-a6a3-e1465c10b8c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:37.040055 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80beda7-9c53-4a23-a6a3-e1465c10b8c6 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:37.040055 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80beda7-9c53-4a23-a6a3-e1465c10b8c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:37.040055 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80beda7-9c53-4a23-a6a3-e1465c10b8c6 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:37.040873 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80beda7-9c53-4a23-a6a3-e1465c10b8c6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:37.040873 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80beda7-9c53-4a23-a6a3-e1465c10b8c6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:37.041352 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80beda7-9c53-4a23-a6a3-e1465c10b8c6 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:37.041530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 490/971] 10.222.0.27 () {60 vars in 1131 bytes} [Wed Sep 7 03:48:36 2022] POST /identity/v3/users => generated 160 bytes in 198 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:37.042910 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0247354-3a1e-40b9-96ce-4a2102566dda None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:37.061993 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 482/972] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:37.067395 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80beda7-9c53-4a23-a6a3-e1465c10b8c6 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:37.094257 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:37.169576 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0247354-3a1e-40b9-96ce-4a2102566dda None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:37.203537 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80beda7-9c53-4a23-a6a3-e1465c10b8c6 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:37.214210 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb92ca14-fd22-4214-a74f-a2ef56c4fd05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:37.215729 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb92ca14-fd22-4214-a74f-a2ef56c4fd05 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:37.216601 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb92ca14-fd22-4214-a74f-a2ef56c4fd05 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:37.217095 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb92ca14-fd22-4214-a74f-a2ef56c4fd05 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:37.230286 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb92ca14-fd22-4214-a74f-a2ef56c4fd05 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:37.233306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb92ca14-fd22-4214-a74f-a2ef56c4fd05 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:37.247061 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa86b24-faed-4786-8d99-cff84ff54972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:37.247781 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa86b24-faed-4786-8d99-cff84ff54972 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:37.248127 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa86b24-faed-4786-8d99-cff84ff54972 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:37.248509 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa86b24-faed-4786-8d99-cff84ff54972 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:37.253220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-eb92ca14-fd22-4214-a74f-a2ef56c4fd05 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 07 03:48:37.253220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:37.253220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:37.253220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:37.253220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:37.253220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:37.253220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:37.253220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:37.253220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:37.253220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:37.253220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:37.253220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:37.253220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:37.253220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:37.253220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:37.253220 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:37.254198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:37.255153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:37.255153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:37.255153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:37.255153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:37.255153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:37.255153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:37.255153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:37.255153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:37.255153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 07 03:48:37.255153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:37.262751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 491/973] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:48:37 2022] GET /identity/v3/users/cinder => generated 84 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:37.265446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa86b24-faed-4786-8d99-cff84ff54972 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:37.265786 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb92ca14-fd22-4214-a74f-a2ef56c4fd05 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:37.267006 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa86b24-faed-4786-8d99-cff84ff54972 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:37.280020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-baa86b24-faed-4786-8d99-cff84ff54972 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 07 03:48:37.280020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:37.280020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:37.280020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:37.280020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:37.280020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:37.280020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:37.280020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:37.280020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:37.280020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:37.280020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:37.280020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:37.280020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:37.280020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:37.280020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:37.280020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:37.281022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:37.281982 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:37.281982 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:37.281982 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:37.281982 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:37.281982 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:37.281982 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:37.281982 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:37.281982 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:37.281982 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 07 03:48:37.281982 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:37.281982 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 483/974] 10.222.0.27 () {56 vars in 1117 bytes} [Wed Sep 7 03:48:37 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:37.286706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa86b24-faed-4786-8d99-cff84ff54972 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:37.292970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:37.381743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb92ca14-fd22-4214-a74f-a2ef56c4fd05 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:37.389885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:37.415518 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa86b24-faed-4786-8d99-cff84ff54972 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:37.418810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf3865c-2dc3-41f4-82a8-cf028e278e12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:37.420059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf3865c-2dc3-41f4-82a8-cf028e278e12 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:37.420059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf3865c-2dc3-41f4-82a8-cf028e278e12 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:37.420059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf3865c-2dc3-41f4-82a8-cf028e278e12 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:37.420622 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf3865c-2dc3-41f4-82a8-cf028e278e12 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:37.420925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf3865c-2dc3-41f4-82a8-cf028e278e12 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:37.422428 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf3865c-2dc3-41f4-82a8-cf028e278e12 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:37.457560 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0954ad-751a-4821-a0db-81771ad2cc9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:37.458642 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0954ad-751a-4821-a0db-81771ad2cc9a None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:37.459174 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0954ad-751a-4821-a0db-81771ad2cc9a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:37.459653 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0954ad-751a-4821-a0db-81771ad2cc9a None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:37.460593 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0954ad-751a-4821-a0db-81771ad2cc9a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:37.461105 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0954ad-751a-4821-a0db-81771ad2cc9a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:37.462837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0954ad-751a-4821-a0db-81771ad2cc9a None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:37.469031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 492/975] 10.222.0.27 () {56 vars in 1101 bytes} [Wed Sep 7 03:48:37 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:37.487966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 484/976] 10.222.0.27 () {56 vars in 1149 bytes} [Wed Sep 7 03:48:37 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:37.493911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf3865c-2dc3-41f4-82a8-cf028e278e12 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:37.591187 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:37.615389 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff6d1b7-7fad-4e06-a9ed-5e1fa9f49392 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:37.615736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff6d1b7-7fad-4e06-a9ed-5e1fa9f49392 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:37.615946 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff6d1b7-7fad-4e06-a9ed-5e1fa9f49392 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:37.618510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf3865c-2dc3-41f4-82a8-cf028e278e12 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:37.618777 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 485/977] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:37.623775 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f4ff6f-1411-4c8e-a268-b2f2321408ba None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:37.624072 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f4ff6f-1411-4c8e-a268-b2f2321408ba None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:37.624338 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f4ff6f-1411-4c8e-a268-b2f2321408ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:37.643580 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-d4f4ff6f-1411-4c8e-a268-b2f2321408ba None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:37.644729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-d4f4ff6f-1411-4c8e-a268-b2f2321408ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:37.656657 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed59995d-0cae-428c-ab2a-7bf536cec583 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:37.657280 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed59995d-0cae-428c-ab2a-7bf536cec583 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:37.657560 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed59995d-0cae-428c-ab2a-7bf536cec583 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:37.657831 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed59995d-0cae-428c-ab2a-7bf536cec583 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:37.734807 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 486/978] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:37 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:37.745969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed59995d-0cae-428c-ab2a-7bf536cec583 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:37.747780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed59995d-0cae-428c-ab2a-7bf536cec583 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:37.826613 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:37.856373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 493/979] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:48:37 2022] GET /identity/v3/auth/tokens => generated 1570 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:48:37.857927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f4ff6f-1411-4c8e-a268-b2f2321408ba None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:37.924594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:37.960913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f4ff6f-1411-4c8e-a268-b2f2321408ba None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:37.994964 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625cd400-1704-43e6-b68e-76b87e41e389 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:37.995600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625cd400-1704-43e6-b68e-76b87e41e389 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:37.995877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625cd400-1704-43e6-b68e-76b87e41e389 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:37.996146 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625cd400-1704-43e6-b68e-76b87e41e389 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:38.006831 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed59995d-0cae-428c-ab2a-7bf536cec583 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:38.007312 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625cd400-1704-43e6-b68e-76b87e41e389 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:38.008537 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625cd400-1704-43e6-b68e-76b87e41e389 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:38.024169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-625cd400-1704-43e6-b68e-76b87e41e389 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:38.024169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:38.024169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:38.024169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:38.024169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:38.024169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:38.024169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:38.024169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:38.024169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:38.024169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:38.024169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:38.024169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:38.024169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:38.024169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:38.024169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:38.024169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:38.025143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:38.026054 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:38.026054 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:38.026054 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:38.026054 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 487/980] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:38.030931 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625cd400-1704-43e6-b68e-76b87e41e389 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:38.108806 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed59995d-0cae-428c-ab2a-7bf536cec583 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:38.138767 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625cd400-1704-43e6-b68e-76b87e41e389 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:38.143393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbd3b1b-cef4-49d7-83e7-225539775bd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:38.144026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbd3b1b-cef4-49d7-83e7-225539775bd2 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:38.144302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbd3b1b-cef4-49d7-83e7-225539775bd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:38.144570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbd3b1b-cef4-49d7-83e7-225539775bd2 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:38.145141 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbd3b1b-cef4-49d7-83e7-225539775bd2 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:38.146544 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbd3b1b-cef4-49d7-83e7-225539775bd2 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-ecbd3b1b-cef4-49d7-83e7-225539775bd2 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:38.155950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:38.157042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:38.157042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:38.157042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:38.157042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:38.157042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:38.157042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:38.157042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:38.157042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:38.157042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:38.157042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:38.157042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:38.157042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:38.157042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:38.157042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:38.157042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:38.157042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:38.158911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:38.158911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:38.158911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:38.158911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:38.158911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:38.158911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:38.158911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:38.158911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:38.158911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:38.158911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:38.158911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:38.158911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:38.158911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 07 03:48:38.158911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:38.158911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 494/981] 10.222.0.27 () {56 vars in 1126 bytes} [Wed Sep 7 03:48:38 2022] GET /identity/v3/services/object-store => generated 93 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:38.159866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:38.162893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbd3b1b-cef4-49d7-83e7-225539775bd2 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:38.180943 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531f46e6-6ecd-455f-9d8f-98dcaf12e5cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:38.181511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531f46e6-6ecd-455f-9d8f-98dcaf12e5cc None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:38.181803 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531f46e6-6ecd-455f-9d8f-98dcaf12e5cc None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:38.182080 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531f46e6-6ecd-455f-9d8f-98dcaf12e5cc None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:38.182834 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531f46e6-6ecd-455f-9d8f-98dcaf12e5cc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:38.183139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531f46e6-6ecd-455f-9d8f-98dcaf12e5cc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:38.185428 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531f46e6-6ecd-455f-9d8f-98dcaf12e5cc None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:38.204679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 488/982] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:38.206260 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531f46e6-6ecd-455f-9d8f-98dcaf12e5cc None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:38.213010 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:38.281643 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbd3b1b-cef4-49d7-83e7-225539775bd2 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:38.319088 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9ac7b9-e53c-4b77-86d9-40f269afc66c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:38.319710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9ac7b9-e53c-4b77-86d9-40f269afc66c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:38.319978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9ac7b9-e53c-4b77-86d9-40f269afc66c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:38.320253 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9ac7b9-e53c-4b77-86d9-40f269afc66c None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:38.320881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9ac7b9-e53c-4b77-86d9-40f269afc66c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:38.321184 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9ac7b9-e53c-4b77-86d9-40f269afc66c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:38.322597 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9ac7b9-e53c-4b77-86d9-40f269afc66c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:38.328229 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 495/983] 10.222.0.27 () {56 vars in 1122 bytes} [Wed Sep 7 03:48:38 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:38.331667 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531f46e6-6ecd-455f-9d8f-98dcaf12e5cc None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:38.334024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9ac7b9-e53c-4b77-86d9-40f269afc66c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:38.368309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32dd1813-2fcd-49c9-b04f-85296fdd689d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:38.368938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dd1813-2fcd-49c9-b04f-85296fdd689d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:38.369227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dd1813-2fcd-49c9-b04f-85296fdd689d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:38.369488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dd1813-2fcd-49c9-b04f-85296fdd689d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:38.377857 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32dd1813-2fcd-49c9-b04f-85296fdd689d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:38.379324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32dd1813-2fcd-49c9-b04f-85296fdd689d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:38.390570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-32dd1813-2fcd-49c9-b04f-85296fdd689d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:38.390570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:38.390570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:38.390570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:38.390570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:38.390570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:38.390570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:38.390570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:38.390570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:38.390570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:38.390570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:38.390570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:38.390570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:38.390570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:38.390570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:38.391559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:38.392438 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:38.392438 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:38.392438 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:38.392438 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:38.392438 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:38.392438 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:38.392438 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:38.392438 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:38.396570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 489/984] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:38.397695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24a79a9-3f01-4925-a48b-dafa7239f2e3 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:38.397984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24a79a9-3f01-4925-a48b-dafa7239f2e3 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:38.398285 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24a79a9-3f01-4925-a48b-dafa7239f2e3 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:38.399536 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 490/985] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:38.400228 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24a79a9-3f01-4925-a48b-dafa7239f2e3 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:38.435134 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9ac7b9-e53c-4b77-86d9-40f269afc66c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:38.437327 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:38.441381 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:38.469531 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ba3aac-8571-42e5-a952-00e0fa50fd2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:38.470171 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ba3aac-8571-42e5-a952-00e0fa50fd2c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:38.470459 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ba3aac-8571-42e5-a952-00e0fa50fd2c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:38.470733 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ba3aac-8571-42e5-a952-00e0fa50fd2c None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:38.471359 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ba3aac-8571-42e5-a952-00e0fa50fd2c None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:38.471667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ba3aac-8571-42e5-a952-00e0fa50fd2c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:38.473038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ba3aac-8571-42e5-a952-00e0fa50fd2c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:38.477773 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 496/986] 10.222.0.27 () {56 vars in 1122 bytes} [Wed Sep 7 03:48:38 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:38.477773 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f49ed9-2d07-4918-ad24-8af7cd270674 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:38.477773 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f49ed9-2d07-4918-ad24-8af7cd270674 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:38.477773 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f49ed9-2d07-4918-ad24-8af7cd270674 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:38.497576 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-52f49ed9-2d07-4918-ad24-8af7cd270674 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:38.499132 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-52f49ed9-2d07-4918-ad24-8af7cd270674 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:38.503884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24a79a9-3f01-4925-a48b-dafa7239f2e3 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:38.537517 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66de5a4d-1f68-46ec-b4a7-5d6968920e07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:38.538139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66de5a4d-1f68-46ec-b4a7-5d6968920e07 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:38.538429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66de5a4d-1f68-46ec-b4a7-5d6968920e07 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:38.538704 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66de5a4d-1f68-46ec-b4a7-5d6968920e07 None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:38.539392 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66de5a4d-1f68-46ec-b4a7-5d6968920e07 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:38.539695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66de5a4d-1f68-46ec-b4a7-5d6968920e07 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:38.541103 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66de5a4d-1f68-46ec-b4a7-5d6968920e07 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:38.552971 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 491/987] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:38.553762 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66de5a4d-1f68-46ec-b4a7-5d6968920e07 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:38.566133 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 497/988] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:38 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:38.566974 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f49ed9-2d07-4918-ad24-8af7cd270674 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:38.663545 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:38.667317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66de5a4d-1f68-46ec-b4a7-5d6968920e07 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:38.673083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:38.678444 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f49ed9-2d07-4918-ad24-8af7cd270674 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:38.705365 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828e2302-7f83-4492-ac79-9428f2ee987e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:38.705896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828e2302-7f83-4492-ac79-9428f2ee987e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:38.706206 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828e2302-7f83-4492-ac79-9428f2ee987e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:38.706542 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828e2302-7f83-4492-ac79-9428f2ee987e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:38.707255 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828e2302-7f83-4492-ac79-9428f2ee987e None admin] RBAC: Adding query filter params (interface=admin, service_id=25aa18d1c2c84ceea0d1e1baa89b7e83, region_id=RegionOne) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:38.707599 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828e2302-7f83-4492-ac79-9428f2ee987e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:38.708961 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828e2302-7f83-4492-ac79-9428f2ee987e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:38.714101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 492/989] 10.222.0.27 () {56 vars in 1249 bytes} [Wed Sep 7 03:48:38 2022] GET /identity/v3/endpoints?service_id=25aa18d1c2c84ceea0d1e1baa89b7e83&interface=admin®ion_id=RegionOne => generated 193 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:38.714798 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9be14b-4d04-4239-8a79-6dc0cdd50092 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:38.715574 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9be14b-4d04-4239-8a79-6dc0cdd50092 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:38.715898 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9be14b-4d04-4239-8a79-6dc0cdd50092 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:38.716216 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9be14b-4d04-4239-8a79-6dc0cdd50092 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:38.716947 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9be14b-4d04-4239-8a79-6dc0cdd50092 None admin] RBAC: Adding query filter params (role.id=d2787a643a3148c882b7a96a59e0a5e3, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=e7b8f511ee884dbfb4f56af78bf75984) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:38.717259 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9be14b-4d04-4239-8a79-6dc0cdd50092 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:38.718737 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9be14b-4d04-4239-8a79-6dc0cdd50092 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:38.720247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828e2302-7f83-4492-ac79-9428f2ee987e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:38.725053 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 498/990] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:48:38 2022] GET /identity/v3/role_assignments?user.id=e7b8f511ee884dbfb4f56af78bf75984&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=d2787a643a3148c882b7a96a59e0a5e3 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:38.837032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828e2302-7f83-4492-ac79-9428f2ee987e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:38.841512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9be14b-4d04-4239-8a79-6dc0cdd50092 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:38.879085 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb837e43-3637-4e1c-b623-d907755353fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:38.879722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb837e43-3637-4e1c-b623-d907755353fc None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:38.880004 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb837e43-3637-4e1c-b623-d907755353fc None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:38.880281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb837e43-3637-4e1c-b623-d907755353fc None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:38.880903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb837e43-3637-4e1c-b623-d907755353fc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:38.882323 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb837e43-3637-4e1c-b623-d907755353fc None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:38.897360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 493/991] 10.222.0.27 () {56 vars in 1087 bytes} [Wed Sep 7 03:48:38 2022] GET /identity/v3/services => generated 1727 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 07 03:48:38.968062 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9be14b-4d04-4239-8a79-6dc0cdd50092 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:39.006878 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b679a6-498d-4bac-8867-83791fc81a4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:39.007525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b679a6-498d-4bac-8867-83791fc81a4d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:39.007806 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b679a6-498d-4bac-8867-83791fc81a4d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:39.008081 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b679a6-498d-4bac-8867-83791fc81a4d None admin] PATH_INFO: `/v3/services/compute` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:39.008686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6b679a6-498d-4bac-8867-83791fc81a4d None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:39.010087 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6b679a6-498d-4bac-8867-83791fc81a4d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-d6b679a6-498d-4bac-8867-83791fc81a4d None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:39.020275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:39.021364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:39.021364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:39.021364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:39.021364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:39.021364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:39.021364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:39.021364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:39.021364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:39.021364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:39.021364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:39.021364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:39.021364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:39.021364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:39.021364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:39.021364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:39.021364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:39.022675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:39.022675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:39.022675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:39.022675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:39.022675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:39.022675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:39.022675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:39.022675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:39.022675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:39.022675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:39.022675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:39.022675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:39.022675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 07 03:48:39.022675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:39.023737 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 499/992] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:38 2022] GET /identity/v3/services/compute => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:39.028398 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb837e43-3637-4e1c-b623-d907755353fc None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:39.059511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:39.141076 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb837e43-3637-4e1c-b623-d907755353fc None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:39.175957 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83594b32-1101-420e-8a4b-cbea184635b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:39.176561 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83594b32-1101-420e-8a4b-cbea184635b7 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:39.176837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83594b32-1101-420e-8a4b-cbea184635b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:39.177106 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83594b32-1101-420e-8a4b-cbea184635b7 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:39.177724 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83594b32-1101-420e-8a4b-cbea184635b7 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:39.178021 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83594b32-1101-420e-8a4b-cbea184635b7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:39.179407 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83594b32-1101-420e-8a4b-cbea184635b7 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:39.184688 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 494/993] 10.222.0.27 () {56 vars in 1112 bytes} [Wed Sep 7 03:48:39 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:39.189934 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b679a6-498d-4bac-8867-83791fc81a4d None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:39.219811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:39.308145 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b679a6-498d-4bac-8867-83791fc81a4d None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:39.344364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609db44f-9c17-463f-896a-b518af2b0dc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:39.344979 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609db44f-9c17-463f-896a-b518af2b0dc5 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:39.345252 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609db44f-9c17-463f-896a-b518af2b0dc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:39.345529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609db44f-9c17-463f-896a-b518af2b0dc5 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:39.346182 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609db44f-9c17-463f-896a-b518af2b0dc5 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:39.346512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609db44f-9c17-463f-896a-b518af2b0dc5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:39.347856 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609db44f-9c17-463f-896a-b518af2b0dc5 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:39.358028 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 500/994] 10.222.0.27 () {56 vars in 1112 bytes} [Wed Sep 7 03:48:39 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:39.359016 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83594b32-1101-420e-8a4b-cbea184635b7 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:39.453182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:39.463902 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83594b32-1101-420e-8a4b-cbea184635b7 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:39.499230 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c2e729-f928-40e2-8e05-2bd41b74bf2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:39.499748 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c2e729-f928-40e2-8e05-2bd41b74bf2d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:39.500036 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c2e729-f928-40e2-8e05-2bd41b74bf2d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:39.500312 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c2e729-f928-40e2-8e05-2bd41b74bf2d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:39.501020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c2e729-f928-40e2-8e05-2bd41b74bf2d None admin] RBAC: Adding query filter params (interface=public, service_id=0e02640f5bab4d569e6c1ecff8db098f, region_id=RegionOne) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:39.501332 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c2e729-f928-40e2-8e05-2bd41b74bf2d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:39.502755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c2e729-f928-40e2-8e05-2bd41b74bf2d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:39.513307 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 495/995] 10.222.0.27 () {56 vars in 1251 bytes} [Wed Sep 7 03:48:39 2022] GET /identity/v3/endpoints?service_id=0e02640f5bab4d569e6c1ecff8db098f&interface=public®ion_id=RegionOne => generated 194 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:39.514534 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609db44f-9c17-463f-896a-b518af2b0dc5 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:39.608256 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:39.619207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609db44f-9c17-463f-896a-b518af2b0dc5 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:39.654804 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8515127a-8a7d-489a-9356-ee9b09865337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:39.655415 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8515127a-8a7d-489a-9356-ee9b09865337 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:39.655690 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8515127a-8a7d-489a-9356-ee9b09865337 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:39.655957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8515127a-8a7d-489a-9356-ee9b09865337 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:39.656555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8515127a-8a7d-489a-9356-ee9b09865337 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:39.657923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8515127a-8a7d-489a-9356-ee9b09865337 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:39.664555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 501/996] 10.222.0.27 () {56 vars in 1087 bytes} [Wed Sep 7 03:48:39 2022] GET /identity/v3/services => generated 1727 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 07 03:48:40.973366 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f162b013-aa58-41cc-86ed-94350551eac3 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:40.973924 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f162b013-aa58-41cc-86ed-94350551eac3 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:40.973924 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f162b013-aa58-41cc-86ed-94350551eac3 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:40.975513 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 496/997] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:40.981919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8874e0c1-260b-4ac4-a56a-6caf29a0b30d None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:40.982496 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8874e0c1-260b-4ac4-a56a-6caf29a0b30d None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:40.982496 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8874e0c1-260b-4ac4-a56a-6caf29a0b30d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:41.001701 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-8874e0c1-260b-4ac4-a56a-6caf29a0b30d None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:41.002938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-8874e0c1-260b-4ac4-a56a-6caf29a0b30d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:41.069139 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 502/998] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:40 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:41.098387 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbc84a7-678c-4542-9ea8-c6a80bae7289 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:41.098675 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbc84a7-678c-4542-9ea8-c6a80bae7289 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:41.098943 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbc84a7-678c-4542-9ea8-c6a80bae7289 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:41.100414 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 497/999] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:41.105351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcec82a3-1a89-44b8-944b-176581697371 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:41.105664 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcec82a3-1a89-44b8-944b-176581697371 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:41.105958 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcec82a3-1a89-44b8-944b-176581697371 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:41.124461 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-bcec82a3-1a89-44b8-944b-176581697371 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:41.125643 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-bcec82a3-1a89-44b8-944b-176581697371 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:41.192669 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 503/1000] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:41 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:41.337675 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbc84a7-678c-4542-9ea8-c6a80bae7289 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:41.441596 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbc84a7-678c-4542-9ea8-c6a80bae7289 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:41.459969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcec82a3-1a89-44b8-944b-176581697371 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:41.475061 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:41.476793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a87859b-3bec-4291-908a-68c0c384e67a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:41.477710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a87859b-3bec-4291-908a-68c0c384e67a None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:41.477996 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a87859b-3bec-4291-908a-68c0c384e67a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:41.478324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a87859b-3bec-4291-908a-68c0c384e67a None admin] PATH_INFO: `/v3/services/image` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:41.478919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a87859b-3bec-4291-908a-68c0c384e67a None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:41.480287 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a87859b-3bec-4291-908a-68c0c384e67a None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-4a87859b-3bec-4291-908a-68c0c384e67a None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:41.489937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:41.491187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:41.491187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:41.491187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:41.491187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:41.491187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:41.491187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:41.491187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:41.491187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:41.491187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:41.491187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:41.491187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:41.491187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:41.491187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:41.491187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:41.491187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:41.491187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:41.492360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:41.492360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:41.492360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:41.492360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:41.492360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:41.492360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:41.492360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:41.492360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:41.492360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:41.492360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:41.492360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:41.492360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:41.492360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 07 03:48:41.492360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:41.495772 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 498/1001] 10.222.0.27 () {56 vars in 1105 bytes} [Wed Sep 7 03:48:41 2022] GET /identity/v3/services/image => generated 86 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:41.497565 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a87859b-3bec-4291-908a-68c0c384e67a None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:41.516980 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:41.569983 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcec82a3-1a89-44b8-944b-176581697371 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:41.608743 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a87859b-3bec-4291-908a-68c0c384e67a None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:41.614083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79e9124-d764-4c6d-a95c-15708c8091a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:41.614748 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79e9124-d764-4c6d-a95c-15708c8091a5 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:41.615031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79e9124-d764-4c6d-a95c-15708c8091a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:41.615307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79e9124-d764-4c6d-a95c-15708c8091a5 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:41.615896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79e9124-d764-4c6d-a95c-15708c8091a5 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:41.617363 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79e9124-d764-4c6d-a95c-15708c8091a5 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-e79e9124-d764-4c6d-a95c-15708c8091a5 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:41.626311 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:41.627306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:41.627306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:41.627306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:41.627306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:41.627306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:41.627306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:41.627306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:41.627306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:41.627306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:41.627306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:41.627306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:41.627306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:41.627306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:41.627306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:41.627306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:41.627306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:41.628240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:41.628240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:41.628240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:41.628240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:41.628240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:41.628240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:41.628240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:41.628240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:41.628240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:41.628240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:41.628240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:41.628240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:41.628240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 07 03:48:41.628240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:41.628240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 504/1002] 10.222.0.27 () {56 vars in 1126 bytes} [Wed Sep 7 03:48:41 2022] GET /identity/v3/services/object-store => generated 93 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:41.633202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79e9124-d764-4c6d-a95c-15708c8091a5 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:41.649702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f87990-500d-4797-97a0-2633557a8b95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:41.650350 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f87990-500d-4797-97a0-2633557a8b95 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:41.650636 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f87990-500d-4797-97a0-2633557a8b95 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:41.650911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f87990-500d-4797-97a0-2633557a8b95 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:41.651548 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f87990-500d-4797-97a0-2633557a8b95 None admin] RBAC: Adding query filter params (name=image) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:41.651879 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f87990-500d-4797-97a0-2633557a8b95 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:41.653781 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f87990-500d-4797-97a0-2633557a8b95 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:41.658916 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 499/1003] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:41 2022] GET /identity/v3/services?name=image => generated 122 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:41.664672 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f87990-500d-4797-97a0-2633557a8b95 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:41.722740 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:41.748741 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79e9124-d764-4c6d-a95c-15708c8091a5 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:41.754637 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:41.774673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f87990-500d-4797-97a0-2633557a8b95 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:41.788113 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed29850-0027-4d22-95ac-ee8fdfedc336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:41.788725 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed29850-0027-4d22-95ac-ee8fdfedc336 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:41.789007 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed29850-0027-4d22-95ac-ee8fdfedc336 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:41.789578 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed29850-0027-4d22-95ac-ee8fdfedc336 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:41.789903 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed29850-0027-4d22-95ac-ee8fdfedc336 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:41.790245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed29850-0027-4d22-95ac-ee8fdfedc336 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:41.791612 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed29850-0027-4d22-95ac-ee8fdfedc336 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:41.799679 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 505/1004] 10.222.0.27 () {56 vars in 1122 bytes} [Wed Sep 7 03:48:41 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:41.805341 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed29850-0027-4d22-95ac-ee8fdfedc336 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:41.817613 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f702980-529b-48ad-800e-1917bdce7b4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:41.818257 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f702980-529b-48ad-800e-1917bdce7b4d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:41.818539 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f702980-529b-48ad-800e-1917bdce7b4d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:41.818815 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f702980-529b-48ad-800e-1917bdce7b4d None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:41.819449 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f702980-529b-48ad-800e-1917bdce7b4d None admin] RBAC: Adding query filter params (type=image) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:41.819750 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f702980-529b-48ad-800e-1917bdce7b4d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:41.821109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f702980-529b-48ad-800e-1917bdce7b4d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:41.825760 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 500/1005] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:41 2022] GET /identity/v3/services?type=image => generated 122 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:41.848195 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6055a7-2758-45e5-8536-e07b18f3f9b0 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:41.848505 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6055a7-2758-45e5-8536-e07b18f3f9b0 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:41.848750 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6055a7-2758-45e5-8536-e07b18f3f9b0 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:41.850174 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 501/1006] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:41.857861 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c9272e-7054-4113-93f1-1d04fb196718 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:41.858187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c9272e-7054-4113-93f1-1d04fb196718 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:41.858455 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c9272e-7054-4113-93f1-1d04fb196718 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:41.878091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-d6c9272e-7054-4113-93f1-1d04fb196718 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:41.879364 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-d6c9272e-7054-4113-93f1-1d04fb196718 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:41.932410 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed29850-0027-4d22-95ac-ee8fdfedc336 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:41.958109 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 502/1007] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:41 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:41.970481 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550d1d06-944d-4c22-b015-062b57e3a220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:41.971367 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550d1d06-944d-4c22-b015-062b57e3a220 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:41.971797 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550d1d06-944d-4c22-b015-062b57e3a220 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:41.972218 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550d1d06-944d-4c22-b015-062b57e3a220 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:41.973016 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550d1d06-944d-4c22-b015-062b57e3a220 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:41.973502 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550d1d06-944d-4c22-b015-062b57e3a220 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:41.975079 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550d1d06-944d-4c22-b015-062b57e3a220 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:41.981909 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 506/1008] 10.222.0.27 () {56 vars in 1122 bytes} [Wed Sep 7 03:48:41 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:41.987782 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c9272e-7054-4113-93f1-1d04fb196718 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:42.067015 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:42.091775 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c9272e-7054-4113-93f1-1d04fb196718 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:42.126790 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40604c18-bd9f-4e3a-8879-e6b1a480bda5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:42.127400 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40604c18-bd9f-4e3a-8879-e6b1a480bda5 None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:42.127679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40604c18-bd9f-4e3a-8879-e6b1a480bda5 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:42.127944 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40604c18-bd9f-4e3a-8879-e6b1a480bda5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:42.128605 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40604c18-bd9f-4e3a-8879-e6b1a480bda5 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:42.130022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40604c18-bd9f-4e3a-8879-e6b1a480bda5 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:42.154140 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 503/1009] 10.222.0.27 () {60 vars in 1144 bytes} [Wed Sep 7 03:48:41 2022] POST /identity/v3/endpoints => generated 325 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 07 03:48:42.233198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550d1d06-944d-4c22-b015-062b57e3a220 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:42.243390 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:42.350266 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550d1d06-944d-4c22-b015-062b57e3a220 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:42.394208 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7639e5-719b-4780-9e41-0aaf1fcfca3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:42.394920 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7639e5-719b-4780-9e41-0aaf1fcfca3e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:42.395199 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7639e5-719b-4780-9e41-0aaf1fcfca3e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:42.395473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7639e5-719b-4780-9e41-0aaf1fcfca3e None admin] PATH_INFO: `/v3/services/compute` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:42.396051 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7639e5-719b-4780-9e41-0aaf1fcfca3e None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:42.397410 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7639e5-719b-4780-9e41-0aaf1fcfca3e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-ad7639e5-719b-4780-9e41-0aaf1fcfca3e None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:42.406454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:42.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:42.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:42.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:42.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:42.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:42.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:42.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:42.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:42.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:42.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:42.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:42.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:42.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:42.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:42.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:42.407455 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:42.408290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:42.408290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:42.408290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:42.408290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:42.408290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:42.408290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:42.408290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:42.408290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:42.408290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:42.408290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:42.408290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:42.408290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:42.408290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 07 03:48:42.408290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:42.408290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 507/1010] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:42 2022] GET /identity/v3/services/compute => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:42.413670 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40604c18-bd9f-4e3a-8879-e6b1a480bda5 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:42.515116 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40604c18-bd9f-4e3a-8879-e6b1a480bda5 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:42.549210 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851d2008-cea3-400e-9c9e-2e4a138ed089 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:42.549824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851d2008-cea3-400e-9c9e-2e4a138ed089 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:42.550105 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851d2008-cea3-400e-9c9e-2e4a138ed089 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:42.550408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851d2008-cea3-400e-9c9e-2e4a138ed089 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:42.551042 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851d2008-cea3-400e-9c9e-2e4a138ed089 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:42.551347 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851d2008-cea3-400e-9c9e-2e4a138ed089 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:42.552734 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851d2008-cea3-400e-9c9e-2e4a138ed089 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:42.554663 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:42.564449 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 504/1011] 10.222.0.27 () {56 vars in 1112 bytes} [Wed Sep 7 03:48:42 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:42.565545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7639e5-719b-4780-9e41-0aaf1fcfca3e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:42.636057 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:42.673708 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7639e5-719b-4780-9e41-0aaf1fcfca3e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:42.707274 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852b0fde-1571-465a-bb9b-cccc410d676f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:42.707888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852b0fde-1571-465a-bb9b-cccc410d676f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:42.708166 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852b0fde-1571-465a-bb9b-cccc410d676f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:42.708438 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852b0fde-1571-465a-bb9b-cccc410d676f None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:42.709062 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852b0fde-1571-465a-bb9b-cccc410d676f None admin] RBAC: Adding query filter params (type=compute) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:42.709366 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852b0fde-1571-465a-bb9b-cccc410d676f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:42.710773 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852b0fde-1571-465a-bb9b-cccc410d676f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:42.721140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 508/1012] 10.222.0.27 () {56 vars in 1112 bytes} [Wed Sep 7 03:48:42 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:42.722503 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851d2008-cea3-400e-9c9e-2e4a138ed089 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:42.825719 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851d2008-cea3-400e-9c9e-2e4a138ed089 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:42.859832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b29814-9284-42a0-bf59-a1ed5ea91afb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:42.860442 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b29814-9284-42a0-bf59-a1ed5ea91afb None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:42.860721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b29814-9284-42a0-bf59-a1ed5ea91afb None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:42.860993 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b29814-9284-42a0-bf59-a1ed5ea91afb None admin] PATH_INFO: `/v3/endpoints` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:42.861652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b29814-9284-42a0-bf59-a1ed5ea91afb None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:42.863031 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b29814-9284-42a0-bf59-a1ed5ea91afb None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:42.884343 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 505/1013] 10.222.0.27 () {60 vars in 1144 bytes} [Wed Sep 7 03:48:42 2022] POST /identity/v3/endpoints => generated 334 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:48:44.020668 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c94c735-94db-4000-b2cf-331d6f85ff1d None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:44.021199 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c94c735-94db-4000-b2cf-331d6f85ff1d None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:44.021199 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c94c735-94db-4000-b2cf-331d6f85ff1d None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:44.022748 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 509/1014] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:44.028032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbb998b-a611-42df-94b1-794ae525a10a None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:44.028592 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbb998b-a611-42df-94b1-794ae525a10a None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:44.028592 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbb998b-a611-42df-94b1-794ae525a10a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:44.047025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-9dbb998b-a611-42df-94b1-794ae525a10a None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:44.048228 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-9dbb998b-a611-42df-94b1-794ae525a10a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:44.102066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:44.127154 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 506/1015] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:44 2022] POST /identity/v3/auth/tokens => generated 1873 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:44.364051 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990ec5e2-9069-4ec1-b41b-39c4a2fffa15 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:44.364404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990ec5e2-9069-4ec1-b41b-39c4a2fffa15 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:44.364613 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990ec5e2-9069-4ec1-b41b-39c4a2fffa15 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:44.366005 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 510/1016] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:44.371727 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7644dc-e89c-4986-8d9e-b7eacbd09e46 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:44.372042 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7644dc-e89c-4986-8d9e-b7eacbd09e46 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:44.372318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7644dc-e89c-4986-8d9e-b7eacbd09e46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:44.391046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-7d7644dc-e89c-4986-8d9e-b7eacbd09e46 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:44.392241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-7d7644dc-e89c-4986-8d9e-b7eacbd09e46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:44.404876 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990ec5e2-9069-4ec1-b41b-39c4a2fffa15 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:44.464650 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 507/1017] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:44 2022] POST /identity/v3/auth/tokens => generated 1873 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:44.513304 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990ec5e2-9069-4ec1-b41b-39c4a2fffa15 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:44.542750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:44.549770 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cfe4b9-df9a-4f2f-a6eb-98a742f8ea93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:44.550409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cfe4b9-df9a-4f2f-a6eb-98a742f8ea93 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:44.550693 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cfe4b9-df9a-4f2f-a6eb-98a742f8ea93 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:44.550969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cfe4b9-df9a-4f2f-a6eb-98a742f8ea93 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:44.551624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cfe4b9-df9a-4f2f-a6eb-98a742f8ea93 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:44.552998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cfe4b9-df9a-4f2f-a6eb-98a742f8ea93 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:44.574156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 511/1018] 10.222.0.27 () {60 vars in 1141 bytes} [Wed Sep 7 03:48:44 2022] POST /identity/v3/services => generated 242 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 07 03:48:44.737608 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7644dc-e89c-4986-8d9e-b7eacbd09e46 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:44.845311 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7644dc-e89c-4986-8d9e-b7eacbd09e46 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:44.859839 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:44.887851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8164c65b-8849-49de-ad64-eef85397170e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:44.888850 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8164c65b-8849-49de-ad64-eef85397170e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:44.889361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8164c65b-8849-49de-ad64-eef85397170e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:44.889841 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8164c65b-8849-49de-ad64-eef85397170e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:44.891187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8164c65b-8849-49de-ad64-eef85397170e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:44.892844 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8164c65b-8849-49de-ad64-eef85397170e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:44.895010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 508/1019] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:44.900403 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cfe4b9-df9a-4f2f-a6eb-98a742f8ea93 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:45.007246 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cfe4b9-df9a-4f2f-a6eb-98a742f8ea93 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:45.041117 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a29d2b-4d94-404a-8e2c-6946778aece0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:45.041778 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a29d2b-4d94-404a-8e2c-6946778aece0 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:45.042005 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a29d2b-4d94-404a-8e2c-6946778aece0 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:45.042294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a29d2b-4d94-404a-8e2c-6946778aece0 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:45.042941 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a29d2b-4d94-404a-8e2c-6946778aece0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:45.045163 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a29d2b-4d94-404a-8e2c-6946778aece0 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:45.074947 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 512/1020] 10.222.0.27 () {60 vars in 1141 bytes} [Wed Sep 7 03:48:44 2022] POST /identity/v3/projects => generated 311 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:48:45.200968 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193bbb63-e246-45d0-b711-84c0dab4df4d None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:45.201504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193bbb63-e246-45d0-b711-84c0dab4df4d None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:45.201504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193bbb63-e246-45d0-b711-84c0dab4df4d None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:45.203203 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 509/1021] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:45.208895 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238de924-6a5a-4199-89fb-8b02ffb62735 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:45.209182 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238de924-6a5a-4199-89fb-8b02ffb62735 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:45.209457 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238de924-6a5a-4199-89fb-8b02ffb62735 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:45.232297 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-238de924-6a5a-4199-89fb-8b02ffb62735 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:45.233525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-238de924-6a5a-4199-89fb-8b02ffb62735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:45.275822 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:45.312524 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 513/1022] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:45 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:45.593586 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193bbb63-e246-45d0-b711-84c0dab4df4d None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:45.697222 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193bbb63-e246-45d0-b711-84c0dab4df4d None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:45.735291 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5233253c-37c0-4868-8232-5673d242edd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:45.735917 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5233253c-37c0-4868-8232-5673d242edd0 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:45.736192 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5233253c-37c0-4868-8232-5673d242edd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:45.736460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5233253c-37c0-4868-8232-5673d242edd0 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:45.746955 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5233253c-37c0-4868-8232-5673d242edd0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:45.751984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-5233253c-37c0-4868-8232-5673d242edd0 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 07 03:48:45.751984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:45.751984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:45.751984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:45.751984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:45.751984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:45.751984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:45.751984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:45.751984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:45.751984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:45.751984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:45.751984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:45.751984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:45.751984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:45.751984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:45.751984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:45.753089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:45.753089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:45.753089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:45.753089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:45.753089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:45.753089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:45.753089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:45.753089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:45.753089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:45.753089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:45.753089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:45.753089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:45.753089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:45.753089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:45.753089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:45.753089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:45.753973 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:45.753973 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:45.753973 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:45.753973 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:45.753973 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:45.753973 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:45.753973 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:45.753973 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:45.753973 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:45.753973 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:45.753973 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 07 03:48:45.753973 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:45.755209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 510/1023] 10.222.0.27 () {56 vars in 1120 bytes} [Wed Sep 7 03:48:45 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:45.758944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238de924-6a5a-4199-89fb-8b02ffb62735 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:45.869143 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238de924-6a5a-4199-89fb-8b02ffb62735 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:45.904681 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c721aae0-f862-4eda-acc9-5995b2c32838 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:45.905294 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c721aae0-f862-4eda-acc9-5995b2c32838 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:45.905570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c721aae0-f862-4eda-acc9-5995b2c32838 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:45.905837 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c721aae0-f862-4eda-acc9-5995b2c32838 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:45.906525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c721aae0-f862-4eda-acc9-5995b2c32838 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:45.906828 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c721aae0-f862-4eda-acc9-5995b2c32838 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:45.908184 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c721aae0-f862-4eda-acc9-5995b2c32838 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:45.921383 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 514/1024] 10.222.0.27 () {56 vars in 1115 bytes} [Wed Sep 7 03:48:45 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:45.922708 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5233253c-37c0-4868-8232-5673d242edd0 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:45.959225 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:46.028347 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5233253c-37c0-4868-8232-5673d242edd0 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:46.062926 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b83991d-9592-4298-bd0d-04fe5d4c1a91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:46.063670 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b83991d-9592-4298-bd0d-04fe5d4c1a91 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:46.063954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b83991d-9592-4298-bd0d-04fe5d4c1a91 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:46.064226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b83991d-9592-4298-bd0d-04fe5d4c1a91 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:46.074023 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b83991d-9592-4298-bd0d-04fe5d4c1a91 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:46.075862 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b83991d-9592-4298-bd0d-04fe5d4c1a91 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:46.090827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-6b83991d-9592-4298-bd0d-04fe5d4c1a91 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:46.090827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:46.090827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:46.090827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:46.090827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:46.090827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:46.090827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:46.090827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:46.090827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:46.090827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:46.090827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:46.090827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:46.090827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:46.090827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:46.090827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:46.090827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:46.091755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:46.097276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:46.097276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:46.097276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:46.097276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 511/1025] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:46.098087 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c721aae0-f862-4eda-acc9-5995b2c32838 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:46.133202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:46.201886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c721aae0-f862-4eda-acc9-5995b2c32838 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:46.240909 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8bae44-b0d5-4f58-b169-ef438c5bbe1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:46.241531 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8bae44-b0d5-4f58-b169-ef438c5bbe1e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:46.241809 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8bae44-b0d5-4f58-b169-ef438c5bbe1e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:46.242079 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8bae44-b0d5-4f58-b169-ef438c5bbe1e None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:46.242747 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8bae44-b0d5-4f58-b169-ef438c5bbe1e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:46.243048 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8bae44-b0d5-4f58-b169-ef438c5bbe1e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:46.244412 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8bae44-b0d5-4f58-b169-ef438c5bbe1e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:46.258018 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 515/1026] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:46.264802 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b83991d-9592-4298-bd0d-04fe5d4c1a91 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:46.358400 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:46.372937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b83991d-9592-4298-bd0d-04fe5d4c1a91 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:46.407583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60ab214-7064-4778-b4e7-9c058c6a6455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:46.408135 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60ab214-7064-4778-b4e7-9c058c6a6455 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:46.408417 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60ab214-7064-4778-b4e7-9c058c6a6455 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:46.408693 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60ab214-7064-4778-b4e7-9c058c6a6455 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:46.416757 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60ab214-7064-4778-b4e7-9c058c6a6455 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:46.418390 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60ab214-7064-4778-b4e7-9c058c6a6455 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:46.430186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-a60ab214-7064-4778-b4e7-9c058c6a6455 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:48:46.430186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:46.430186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:46.430186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:46.430186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:46.430186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:46.430186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:46.430186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:46.430186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:46.430186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:46.430186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:46.430186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:46.430186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:46.430186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:46.430186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:46.430186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:46.431178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:46.432139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:46.432139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:46.432139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:46.432139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:46.432139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:46.432139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:46.432139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:46.432139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:46.432139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:48:46.432139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:46.433329 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 512/1027] 10.222.0.27 () {56 vars in 1093 bytes} [Wed Sep 7 03:48:46 2022] GET /identity/v3/users/nova => generated 82 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:46.437952 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8bae44-b0d5-4f58-b169-ef438c5bbe1e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:46.537419 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:46.545744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8bae44-b0d5-4f58-b169-ef438c5bbe1e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:46.581664 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f065590-859f-4987-8d29-82d0812e5990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:46.582299 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f065590-859f-4987-8d29-82d0812e5990 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:46.582580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f065590-859f-4987-8d29-82d0812e5990 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:46.582852 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f065590-859f-4987-8d29-82d0812e5990 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:46.583580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f065590-859f-4987-8d29-82d0812e5990 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:46.583881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f065590-859f-4987-8d29-82d0812e5990 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:46.585314 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f065590-859f-4987-8d29-82d0812e5990 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:46.607391 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 516/1028] 10.222.0.27 () {56 vars in 1133 bytes} [Wed Sep 7 03:48:46 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:46.608667 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60ab214-7064-4778-b4e7-9c058c6a6455 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:46.715556 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60ab214-7064-4778-b4e7-9c058c6a6455 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:46.749118 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e287605-986f-42de-b9b0-592a50e3778d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:46.749724 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e287605-986f-42de-b9b0-592a50e3778d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:46.750003 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e287605-986f-42de-b9b0-592a50e3778d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:46.750307 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e287605-986f-42de-b9b0-592a50e3778d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:46.768731 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:46.794288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4b93b6-5eb4-42da-9ad7-d54150933ba3 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:46.794608 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4b93b6-5eb4-42da-9ad7-d54150933ba3 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:46.794855 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4b93b6-5eb4-42da-9ad7-d54150933ba3 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:46.796344 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 517/1029] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:48:46.801240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da2dbad-bcff-46ff-96bc-e492ffecaa79 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:46.801520 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da2dbad-bcff-46ff-96bc-e492ffecaa79 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:46.801798 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da2dbad-bcff-46ff-96bc-e492ffecaa79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:46.820606 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-0da2dbad-bcff-46ff-96bc-e492ffecaa79 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:46.820900 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e287605-986f-42de-b9b0-592a50e3778d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:46.821806 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-0da2dbad-bcff-46ff-96bc-e492ffecaa79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:46.822304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e287605-986f-42de-b9b0-592a50e3778d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:46.891417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 518/1030] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:46 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:48:46.929621 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 513/1031] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:48:46 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:48:46.931065 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da2dbad-bcff-46ff-96bc-e492ffecaa79 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:47.015363 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:47.035345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da2dbad-bcff-46ff-96bc-e492ffecaa79 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:47.069762 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604fa74b-3705-48c5-87c5-72e7d26bfd97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:47.070420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604fa74b-3705-48c5-87c5-72e7d26bfd97 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:47.070698 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604fa74b-3705-48c5-87c5-72e7d26bfd97 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:47.070966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604fa74b-3705-48c5-87c5-72e7d26bfd97 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:47.081781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604fa74b-3705-48c5-87c5-72e7d26bfd97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:47.083285 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604fa74b-3705-48c5-87c5-72e7d26bfd97 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:47.097775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-604fa74b-3705-48c5-87c5-72e7d26bfd97 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:47.097775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:47.097775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:47.097775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:47.097775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:47.097775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:47.097775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:47.097775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:47.097775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:47.097775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:47.097775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:47.097775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:47.097775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:47.097775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:47.097775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:47.097775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:47.099284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:47.100198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:47.100198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:47.100198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:47.102436 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 519/1032] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:47.104492 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e287605-986f-42de-b9b0-592a50e3778d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:47.153097 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:47.162871 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604fa74b-3705-48c5-87c5-72e7d26bfd97 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:47.214453 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e287605-986f-42de-b9b0-592a50e3778d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:47.251464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e1a038-6246-4d98-af50-79b0da5e2e08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:47.252076 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e1a038-6246-4d98-af50-79b0da5e2e08 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:47.252352 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e1a038-6246-4d98-af50-79b0da5e2e08 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:47.252624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e1a038-6246-4d98-af50-79b0da5e2e08 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:47.253270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e1a038-6246-4d98-af50-79b0da5e2e08 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:47.253578 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e1a038-6246-4d98-af50-79b0da5e2e08 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:47.257475 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e1a038-6246-4d98-af50-79b0da5e2e08 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:47.269693 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 514/1033] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:47.275164 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e1a038-6246-4d98-af50-79b0da5e2e08 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:47.289398 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604fa74b-3705-48c5-87c5-72e7d26bfd97 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:47.325686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c94b64d-74d9-4c18-8e2e-e7897db36e37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:47.326378 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c94b64d-74d9-4c18-8e2e-e7897db36e37 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:47.326666 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c94b64d-74d9-4c18-8e2e-e7897db36e37 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:47.326941 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c94b64d-74d9-4c18-8e2e-e7897db36e37 None admin] PATH_INFO: `/v3/services/image` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:47.327523 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c94b64d-74d9-4c18-8e2e-e7897db36e37 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:47.328894 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c94b64d-74d9-4c18-8e2e-e7897db36e37 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:47.330989 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-5c94b64d-74d9-4c18-8e2e-e7897db36e37 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:47.339119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:47.340200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:47.340200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:47.340200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:47.340200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:47.340200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:47.340200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:47.340200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:47.340200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:47.340200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:47.340200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:47.340200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:47.340200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:47.340200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:47.340200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:47.340200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:47.340200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:47.341030 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:47.341030 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:47.341030 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:47.341030 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:47.341030 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:47.341030 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:47.341030 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:47.341030 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:47.341030 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:47.341030 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:47.341030 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:47.341030 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:47.341030 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 07 03:48:47.341030 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:47.343323 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 520/1034] 10.222.0.27 () {56 vars in 1105 bytes} [Wed Sep 7 03:48:47 2022] GET /identity/v3/services/image => generated 86 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:47.344565 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dd0762-3f6c-4461-be04-9e41dee9e942 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:47.344850 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dd0762-3f6c-4461-be04-9e41dee9e942 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:47.345125 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dd0762-3f6c-4461-be04-9e41dee9e942 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:47.346460 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 521/1035] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:47.348950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1dd0762-3f6c-4461-be04-9e41dee9e942 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:47.382066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e1a038-6246-4d98-af50-79b0da5e2e08 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:47.391636 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:47.416214 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014c0119-a929-4405-8c8a-64ce9285b12e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:47.416866 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014c0119-a929-4405-8c8a-64ce9285b12e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:47.417148 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014c0119-a929-4405-8c8a-64ce9285b12e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:47.417419 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014c0119-a929-4405-8c8a-64ce9285b12e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:47.423990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014c0119-a929-4405-8c8a-64ce9285b12e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:47.425412 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014c0119-a929-4405-8c8a-64ce9285b12e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:47.437275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-014c0119-a929-4405-8c8a-64ce9285b12e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:47.437275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:47.437275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:47.437275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:47.437275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:47.437275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:47.437275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:47.437275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:47.437275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:47.437275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:47.437275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:47.437275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:47.437275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:47.437275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:47.437275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:47.438361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:47.439464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:47.439464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:47.439464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:47.439464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:47.439464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:47.439464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:47.439464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:47.439464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:47.439464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 515/1036] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:47.440069 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf2aa0b-2406-4da3-bffd-d78d8c97098f None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:47.440069 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf2aa0b-2406-4da3-bffd-d78d8c97098f None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:47.440348 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf2aa0b-2406-4da3-bffd-d78d8c97098f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:47.454805 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1dd0762-3f6c-4461-be04-9e41dee9e942 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:47.460555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-abf2aa0b-2406-4da3-bffd-d78d8c97098f None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:47.461756 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-abf2aa0b-2406-4da3-bffd-d78d8c97098f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:47.489893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a39734-d553-45f0-99cf-31c47d23a309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:47.490534 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a39734-d553-45f0-99cf-31c47d23a309 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:47.490814 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a39734-d553-45f0-99cf-31c47d23a309 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:47.491084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a39734-d553-45f0-99cf-31c47d23a309 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:47.491708 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a39734-d553-45f0-99cf-31c47d23a309 None admin] RBAC: Adding query filter params (name=image) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:47.492009 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a39734-d553-45f0-99cf-31c47d23a309 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:47.493369 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a39734-d553-45f0-99cf-31c47d23a309 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:47.498874 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 522/1037] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:47 2022] GET /identity/v3/services?name=image => generated 122 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:47.499652 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a39734-d553-45f0-99cf-31c47d23a309 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:47.529266 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 516/1038] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:47 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:47.530030 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf2aa0b-2406-4da3-bffd-d78d8c97098f None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:47.567857 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:47.611103 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a39734-d553-45f0-99cf-31c47d23a309 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:47.631328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:47.635464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf2aa0b-2406-4da3-bffd-d78d8c97098f None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:47.647552 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181cf8ac-2de5-449e-be48-18d9034c21d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:47.648169 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181cf8ac-2de5-449e-be48-18d9034c21d4 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:47.648451 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181cf8ac-2de5-449e-be48-18d9034c21d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:47.648722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181cf8ac-2de5-449e-be48-18d9034c21d4 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:47.649475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181cf8ac-2de5-449e-be48-18d9034c21d4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:47.649788 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181cf8ac-2de5-449e-be48-18d9034c21d4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:47.651254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181cf8ac-2de5-449e-be48-18d9034c21d4 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:47.664303 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 523/1039] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:47.670239 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181cf8ac-2de5-449e-be48-18d9034c21d4 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:47.673588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba07d4e9-45f2-45a8-b3d6-862505183649 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:47.674334 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba07d4e9-45f2-45a8-b3d6-862505183649 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:47.674713 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba07d4e9-45f2-45a8-b3d6-862505183649 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:47.675092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba07d4e9-45f2-45a8-b3d6-862505183649 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:47.675823 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba07d4e9-45f2-45a8-b3d6-862505183649 None admin] RBAC: Adding query filter params (type=image) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:47.676230 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba07d4e9-45f2-45a8-b3d6-862505183649 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:47.677701 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba07d4e9-45f2-45a8-b3d6-862505183649 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:47.687829 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 517/1040] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:47 2022] GET /identity/v3/services?type=image => generated 350 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:47.688873 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba07d4e9-45f2-45a8-b3d6-862505183649 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:47.780924 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181cf8ac-2de5-449e-be48-18d9034c21d4 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:47.793352 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba07d4e9-45f2-45a8-b3d6-862505183649 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:47.804161 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:47.816021 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f3d72a-b9e1-4923-9143-42106d2db481 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:47.816628 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f3d72a-b9e1-4923-9143-42106d2db481 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:47.816901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f3d72a-b9e1-4923-9143-42106d2db481 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:47.817179 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f3d72a-b9e1-4923-9143-42106d2db481 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:47.817889 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f3d72a-b9e1-4923-9143-42106d2db481 None admin] RBAC: Adding query filter params (role.id=d2787a643a3148c882b7a96a59e0a5e3, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=953cb6c24bcc4551b91ce540d18cd751) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:47.818219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f3d72a-b9e1-4923-9143-42106d2db481 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:47.819636 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f3d72a-b9e1-4923-9143-42106d2db481 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:47.825777 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 524/1041] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:48:47 2022] GET /identity/v3/role_assignments?user.id=953cb6c24bcc4551b91ce540d18cd751&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=d2787a643a3148c882b7a96a59e0a5e3 => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:47.826528 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f3d72a-b9e1-4923-9143-42106d2db481 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:47.828617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ec612a-b433-41f6-a0fb-1b1e75a83a0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:47.829502 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ec612a-b433-41f6-a0fb-1b1e75a83a0d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:47.829781 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ec612a-b433-41f6-a0fb-1b1e75a83a0d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:47.830054 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ec612a-b433-41f6-a0fb-1b1e75a83a0d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:47.830798 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ec612a-b433-41f6-a0fb-1b1e75a83a0d None admin] RBAC: Adding query filter params (interface=public, service_id=626f6bac88894192a4c305de11aed9c7, region_id=RegionOne) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:47.831112 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ec612a-b433-41f6-a0fb-1b1e75a83a0d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:47.832499 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ec612a-b433-41f6-a0fb-1b1e75a83a0d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:47.848282 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 518/1042] 10.222.0.27 () {56 vars in 1251 bytes} [Wed Sep 7 03:48:47 2022] GET /identity/v3/endpoints?service_id=626f6bac88894192a4c305de11aed9c7&interface=public®ion_id=RegionOne => generated 194 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:47.849955 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ec612a-b433-41f6-a0fb-1b1e75a83a0d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:47.883029 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:47.955721 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f3d72a-b9e1-4923-9143-42106d2db481 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:47.967760 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ec612a-b433-41f6-a0fb-1b1e75a83a0d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:47.993742 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa76526-dac9-4777-a0ed-5bb34370c021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:47.994415 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa76526-dac9-4777-a0ed-5bb34370c021 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:47.994693 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa76526-dac9-4777-a0ed-5bb34370c021 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:47.994964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa76526-dac9-4777-a0ed-5bb34370c021 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:47.996790 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa76526-dac9-4777-a0ed-5bb34370c021 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:47.998387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa76526-dac9-4777-a0ed-5bb34370c021 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:47.999989 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 525/1043] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:48.005550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa76526-dac9-4777-a0ed-5bb34370c021 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:48.007270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10414890-a50e-4a43-8248-e84a5179e65e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:48.007887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10414890-a50e-4a43-8248-e84a5179e65e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:48.008174 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10414890-a50e-4a43-8248-e84a5179e65e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:48.008454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10414890-a50e-4a43-8248-e84a5179e65e None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:48.009067 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10414890-a50e-4a43-8248-e84a5179e65e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:48.010470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10414890-a50e-4a43-8248-e84a5179e65e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:48.017893 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 519/1044] 10.222.0.27 () {56 vars in 1087 bytes} [Wed Sep 7 03:48:47 2022] GET /identity/v3/services => generated 1957 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 07 03:48:48.078179 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:48.128593 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa76526-dac9-4777-a0ed-5bb34370c021 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:48.165217 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496291ec-bc0b-42a4-a915-2045e86de681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:48.165843 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496291ec-bc0b-42a4-a915-2045e86de681 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:48.166149 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496291ec-bc0b-42a4-a915-2045e86de681 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:48.166419 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496291ec-bc0b-42a4-a915-2045e86de681 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:48.167093 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496291ec-bc0b-42a4-a915-2045e86de681 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:48.168738 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496291ec-bc0b-42a4-a915-2045e86de681 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:48.193907 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 526/1045] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:48:48 2022] POST /identity/v3/users => generated 285 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:48:49.996510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7638dc-b5fd-498e-83d5-7fde24737e45 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:49.997036 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7638dc-b5fd-498e-83d5-7fde24737e45 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:49.997036 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7638dc-b5fd-498e-83d5-7fde24737e45 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:49.998613 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 520/1046] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:50.003853 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8550f6-359e-4531-a8b1-508f6437638f None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:50.004267 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8550f6-359e-4531-a8b1-508f6437638f None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:50.004436 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8550f6-359e-4531-a8b1-508f6437638f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:50.023570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-ab8550f6-359e-4531-a8b1-508f6437638f None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:50.024766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-ab8550f6-359e-4531-a8b1-508f6437638f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:50.096109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 527/1047] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:50 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:48:50.206600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3af8c4-d602-46f2-8cb5-19beec0f91b2 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:50.206922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3af8c4-d602-46f2-8cb5-19beec0f91b2 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:50.207161 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3af8c4-d602-46f2-8cb5-19beec0f91b2 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:50.208638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 521/1048] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:50.213628 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25371071-071d-4f73-99eb-b103e07dc529 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:50.213924 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25371071-071d-4f73-99eb-b103e07dc529 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:50.214236 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25371071-071d-4f73-99eb-b103e07dc529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:50.232886 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-25371071-071d-4f73-99eb-b103e07dc529 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:50.234610 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-25371071-071d-4f73-99eb-b103e07dc529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:50.273901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:50.312163 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 528/1049] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:50 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:50.363181 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3af8c4-d602-46f2-8cb5-19beec0f91b2 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:50.422808 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5fbf44-b69b-4647-962c-818f3acc2d8b None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:50.423153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5fbf44-b69b-4647-962c-818f3acc2d8b None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:50.423361 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5fbf44-b69b-4647-962c-818f3acc2d8b None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:50.424845 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 529/1050] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:48:50.429497 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87b82ee-4e17-4b17-aff3-a5c7494f13d0 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:50.429779 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87b82ee-4e17-4b17-aff3-a5c7494f13d0 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:50.430053 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87b82ee-4e17-4b17-aff3-a5c7494f13d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:50.448512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-f87b82ee-4e17-4b17-aff3-a5c7494f13d0 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:50.449741 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-f87b82ee-4e17-4b17-aff3-a5c7494f13d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:50.462414 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:50.474858 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3af8c4-d602-46f2-8cb5-19beec0f91b2 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:50.511091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac128700-abcb-4df9-9c41-bd9478138346 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:50.512190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac128700-abcb-4df9-9c41-bd9478138346 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:50.512520 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac128700-abcb-4df9-9c41-bd9478138346 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:50.512784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac128700-abcb-4df9-9c41-bd9478138346 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:50.520726 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 530/1051] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:50 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:48:50.522907 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac128700-abcb-4df9-9c41-bd9478138346 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:50.528276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-ac128700-abcb-4df9-9c41-bd9478138346 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 07 03:48:50.528276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:50.528276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:50.528276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:50.528276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:50.528276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:50.528276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:50.528276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:50.528276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:50.528276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:50.528276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:50.528276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:50.528276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:50.528276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:50.528276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:50.528276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:50.529212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:50.529212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:50.529212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:50.529212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:50.529212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:50.529212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:50.529212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:50.529212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:50.529212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:50.529212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:50.529212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:50.529212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:50.529212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:50.529212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:50.529212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:50.529212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:50.530066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:50.530066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:50.530066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:50.530066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:50.530066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:50.530066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:50.530066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:50.530066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:50.530066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:50.530066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:50.530066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 07 03:48:50.530066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:50.530066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 522/1052] 10.222.0.27 () {56 vars in 1120 bytes} [Wed Sep 7 03:48:50 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:50.535327 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87b82ee-4e17-4b17-aff3-a5c7494f13d0 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:50.588258 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac128700-abcb-4df9-9c41-bd9478138346 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:50.638243 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:50.642765 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87b82ee-4e17-4b17-aff3-a5c7494f13d0 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:50.685634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4fc736-19b3-462a-9d75-cd3c2f9cc4ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:50.686570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4fc736-19b3-462a-9d75-cd3c2f9cc4ad None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:50.686854 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4fc736-19b3-462a-9d75-cd3c2f9cc4ad None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:50.687238 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4fc736-19b3-462a-9d75-cd3c2f9cc4ad None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:50.688154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4fc736-19b3-462a-9d75-cd3c2f9cc4ad None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:50.688609 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4fc736-19b3-462a-9d75-cd3c2f9cc4ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:50.690550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4fc736-19b3-462a-9d75-cd3c2f9cc4ad None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:50.692275 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac128700-abcb-4df9-9c41-bd9478138346 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:50.704352 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 531/1053] 10.222.0.27 () {56 vars in 1115 bytes} [Wed Sep 7 03:48:50 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:50.706213 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4fc736-19b3-462a-9d75-cd3c2f9cc4ad None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:50.733391 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a78fba-4893-4c68-8537-8624585fc2f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:50.733885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a78fba-4893-4c68-8537-8624585fc2f7 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:50.734187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a78fba-4893-4c68-8537-8624585fc2f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:50.734503 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a78fba-4893-4c68-8537-8624585fc2f7 None admin] PATH_INFO: `/v3/services/image` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:50.735031 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a78fba-4893-4c68-8537-8624585fc2f7 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:50.736370 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a78fba-4893-4c68-8537-8624585fc2f7 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-47a78fba-4893-4c68-8537-8624585fc2f7 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:50.745317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:50.746659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:50.746659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:50.746659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:50.746659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:50.746659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:50.746659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:50.746659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:48:50.746659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:48:50.746659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:48:50.746659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:48:50.746659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:50.746659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:50.746659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:50.746659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:50.746659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:50.746659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:50.751629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:50.751629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:50.751629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:50.751629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:50.751629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:50.751629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:50.751629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:50.751629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:50.751629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:50.751629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:50.751629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:48:50.751629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:48:50.751629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 07 03:48:50.751629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:50.751629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 523/1054] 10.222.0.27 () {56 vars in 1105 bytes} [Wed Sep 7 03:48:50 2022] GET /identity/v3/services/image => generated 86 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:50.752641 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a78fba-4893-4c68-8537-8624585fc2f7 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:50.768880 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:50.819392 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4fc736-19b3-462a-9d75-cd3c2f9cc4ad None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:50.853135 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46f378e-eb4d-420b-9847-056c8f916372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:50.853762 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46f378e-eb4d-420b-9847-056c8f916372 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:50.854031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46f378e-eb4d-420b-9847-056c8f916372 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:50.854322 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46f378e-eb4d-420b-9847-056c8f916372 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:50.855575 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a78fba-4893-4c68-8537-8624585fc2f7 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:50.864772 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46f378e-eb4d-420b-9847-056c8f916372 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:50.866249 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46f378e-eb4d-420b-9847-056c8f916372 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:50.880881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-f46f378e-eb4d-420b-9847-056c8f916372 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:50.880881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:50.880881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:50.880881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:50.880881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:50.880881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:50.880881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:50.880881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:50.880881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:50.880881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:50.880881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:50.880881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:50.880881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:50.880881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:50.880881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:50.880881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:50.881998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:50.883141 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:50.883141 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:50.883141 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:50.883141 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 532/1055] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:50.883141 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46f378e-eb4d-420b-9847-056c8f916372 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:50.893105 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3856f1-139b-4178-b733-59429ce9383c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:50.893718 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3856f1-139b-4178-b733-59429ce9383c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:50.894000 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3856f1-139b-4178-b733-59429ce9383c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:50.894303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3856f1-139b-4178-b733-59429ce9383c None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:50.894937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3856f1-139b-4178-b733-59429ce9383c None admin] RBAC: Adding query filter params (name=image) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:50.895237 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3856f1-139b-4178-b733-59429ce9383c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:50.896590 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3856f1-139b-4178-b733-59429ce9383c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:50.899136 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:50.901325 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 524/1056] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:50 2022] GET /identity/v3/services?name=image => generated 122 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:50.902098 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3856f1-139b-4178-b733-59429ce9383c None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:50.978686 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:50.989953 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46f378e-eb4d-420b-9847-056c8f916372 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:51.001180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3856f1-139b-4178-b733-59429ce9383c None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:51.029403 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab423e6-5602-420e-b6c1-0a164bc3be94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:51.029965 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab423e6-5602-420e-b6c1-0a164bc3be94 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:51.030278 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab423e6-5602-420e-b6c1-0a164bc3be94 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:51.030565 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab423e6-5602-420e-b6c1-0a164bc3be94 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:51.035248 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f619099-7035-4cdb-860b-a9d4adfe288a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:51.035848 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f619099-7035-4cdb-860b-a9d4adfe288a None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:51.036120 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f619099-7035-4cdb-860b-a9d4adfe288a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:51.036392 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f619099-7035-4cdb-860b-a9d4adfe288a None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:51.037024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f619099-7035-4cdb-860b-a9d4adfe288a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:51.037318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f619099-7035-4cdb-860b-a9d4adfe288a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:51.038702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f619099-7035-4cdb-860b-a9d4adfe288a None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:51.042176 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab423e6-5602-420e-b6c1-0a164bc3be94 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:51.048177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-4ab423e6-5602-420e-b6c1-0a164bc3be94 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:51.048177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:51.048177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:51.048177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:51.048177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:51.048177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:51.048177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:51.048177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:51.048177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:51.048177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:51.048177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:51.048177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:51.048177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:51.048177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:51.048177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:51.048177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:51.049197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:51.049197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:51.049197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:51.049197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:51.049197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:51.049197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:51.049197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:51.049197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:51.049197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:51.049197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:51.049197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:51.049197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:51.049197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:51.049197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:51.049197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:51.049197 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:51.050067 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:51.050067 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:51.050067 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:51.050067 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:51.050067 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:51.050067 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:51.050067 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:51.050067 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:51.050067 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:51.050067 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:51.050067 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:51.050067 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:51.050067 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 533/1057] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:48:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:51.050806 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab423e6-5602-420e-b6c1-0a164bc3be94 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:51.051124 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 525/1058] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:51.055663 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f619099-7035-4cdb-860b-a9d4adfe288a None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:51.130368 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:51.155035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f619099-7035-4cdb-860b-a9d4adfe288a None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:51.155582 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab423e6-5602-420e-b6c1-0a164bc3be94 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:51.188887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26159488-443d-45c4-bb8c-e870375bd8d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:51.189485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26159488-443d-45c4-bb8c-e870375bd8d8 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:51.189716 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da06632-543d-44fb-b029-26a849f38213 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:51.189827 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26159488-443d-45c4-bb8c-e870375bd8d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:51.190021 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26159488-443d-45c4-bb8c-e870375bd8d8 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:51.190352 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da06632-543d-44fb-b029-26a849f38213 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:51.190639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da06632-543d-44fb-b029-26a849f38213 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:51.190730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26159488-443d-45c4-bb8c-e870375bd8d8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:51.190917 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da06632-543d-44fb-b029-26a849f38213 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:51.191006 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26159488-443d-45c4-bb8c-e870375bd8d8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:51.191543 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da06632-543d-44fb-b029-26a849f38213 None admin] RBAC: Adding query filter params (type=image) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:51.191850 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da06632-543d-44fb-b029-26a849f38213 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:51.192349 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26159488-443d-45c4-bb8c-e870375bd8d8 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:51.193208 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da06632-543d-44fb-b029-26a849f38213 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:51.197262 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:51.198254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 534/1059] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:51 2022] GET /identity/v3/services?type=image => generated 350 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:51.199062 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da06632-543d-44fb-b029-26a849f38213 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:51.206151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 526/1060] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:48:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:51.206151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26159488-443d-45c4-bb8c-e870375bd8d8 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:51.305257 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26159488-443d-45c4-bb8c-e870375bd8d8 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:51.305844 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da06632-543d-44fb-b029-26a849f38213 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:51.341351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e8c58d-dcf3-43e4-aa0c-59b365ed93ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:51.341977 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e8c58d-dcf3-43e4-aa0c-59b365ed93ff None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:51.342288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e8c58d-dcf3-43e4-aa0c-59b365ed93ff None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:51.342388 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c448a66f-ae01-4708-8902-fe51569a96f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:51.342560 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e8c58d-dcf3-43e4-aa0c-59b365ed93ff None admin] PATH_INFO: `/v3/users/nova` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:51.342901 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c448a66f-ae01-4708-8902-fe51569a96f5 None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:51.343183 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c448a66f-ae01-4708-8902-fe51569a96f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:51.343462 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c448a66f-ae01-4708-8902-fe51569a96f5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:51.344124 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c448a66f-ae01-4708-8902-fe51569a96f5 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:51.345490 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c448a66f-ae01-4708-8902-fe51569a96f5 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:51.350208 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e8c58d-dcf3-43e4-aa0c-59b365ed93ff None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:51.351664 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e8c58d-dcf3-43e4-aa0c-59b365ed93ff None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:51.353387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:51.363957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-d8e8c58d-dcf3-43e4-aa0c-59b365ed93ff None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:48:51.363957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:51.363957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:51.363957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:51.363957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:51.363957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:51.363957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:51.363957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:51.363957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:51.363957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:51.363957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:51.363957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:51.363957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:51.363957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:51.363957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:51.363957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:51.365154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:51.377942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:51.377942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:51.377942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:51.377942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:51.377942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:51.377942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:51.377942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:51.377942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:51.377942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:48:51.377942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:51.377942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 535/1061] 10.222.0.27 () {56 vars in 1093 bytes} [Wed Sep 7 03:48:51 2022] GET /identity/v3/users/nova => generated 82 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:51.378678 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 527/1062] 10.222.0.27 () {60 vars in 1144 bytes} [Wed Sep 7 03:48:51 2022] POST /identity/v3/endpoints => generated 327 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 07 03:48:51.378758 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e8c58d-dcf3-43e4-aa0c-59b365ed93ff None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:51.378828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c448a66f-ae01-4708-8902-fe51569a96f5 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:51.442342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:51.480767 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e8c58d-dcf3-43e4-aa0c-59b365ed93ff None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:51.503131 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c448a66f-ae01-4708-8902-fe51569a96f5 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:51.518727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3104b2-44c8-44df-8996-73d22d64ba9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:51.519396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3104b2-44c8-44df-8996-73d22d64ba9c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:51.519678 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3104b2-44c8-44df-8996-73d22d64ba9c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:51.519982 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3104b2-44c8-44df-8996-73d22d64ba9c None admin] PATH_INFO: `/v3/users/b86b7649345244fd8fcd3a80e3024209` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:51.536331 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3104b2-44c8-44df-8996-73d22d64ba9c None admin] RBAC: Authorizing `identity:get_user(user_id=b86b7649345244fd8fcd3a80e3024209)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:51.537790 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3104b2-44c8-44df-8996-73d22d64ba9c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:51.539532 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 536/1063] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:48:51 2022] GET /identity/v3/users/b86b7649345244fd8fcd3a80e3024209 => generated 285 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:51.545401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3104b2-44c8-44df-8996-73d22d64ba9c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:51.547458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b539b021-35e0-4078-b972-b62b58897032 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:51.547458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b539b021-35e0-4078-b972-b62b58897032 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:51.547458 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b539b021-35e0-4078-b972-b62b58897032 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:51.547734 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b539b021-35e0-4078-b972-b62b58897032 None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:51.548432 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b539b021-35e0-4078-b972-b62b58897032 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:51.549169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b539b021-35e0-4078-b972-b62b58897032 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:51.550176 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b539b021-35e0-4078-b972-b62b58897032 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:51.572178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 528/1064] 10.222.0.27 () {56 vars in 1133 bytes} [Wed Sep 7 03:48:51 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:51.579256 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b539b021-35e0-4078-b972-b62b58897032 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:51.609549 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:51.657478 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3104b2-44c8-44df-8996-73d22d64ba9c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:51.691317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b539b021-35e0-4078-b972-b62b58897032 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:51.692874 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2b73ea-0f27-4585-906a-a328b30441cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:51.693485 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2b73ea-0f27-4585-906a-a328b30441cc None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:51.693766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2b73ea-0f27-4585-906a-a328b30441cc None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:51.694041 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2b73ea-0f27-4585-906a-a328b30441cc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:51.703149 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:51.728837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6aa335c-6af5-4616-9731-66b81d49e71d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:51.729475 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aa335c-6af5-4616-9731-66b81d49e71d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:51.729755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aa335c-6af5-4616-9731-66b81d49e71d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:51.730027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aa335c-6af5-4616-9731-66b81d49e71d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:51.741070 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6aa335c-6af5-4616-9731-66b81d49e71d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:51.742559 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6aa335c-6af5-4616-9731-66b81d49e71d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:51.757555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-f6aa335c-6af5-4616-9731-66b81d49e71d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:51.757555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:51.757555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:51.757555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:51.757555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:51.757555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:51.757555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:51.757555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:51.757555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:51.757555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:51.757555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:51.757555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:51.757555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:51.757555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:51.757555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:51.757555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:51.758546 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:51.759499 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:51.759499 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:51.759499 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:51.763630 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 529/1065] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:51.764363 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6aa335c-6af5-4616-9731-66b81d49e71d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:51.767725 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2b73ea-0f27-4585-906a-a328b30441cc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:51.769182 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2b73ea-0f27-4585-906a-a328b30441cc None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:51.827186 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:51.870321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6aa335c-6af5-4616-9731-66b81d49e71d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:51.880233 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 537/1066] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:48:51 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 07 03:48:51.881699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2b73ea-0f27-4585-906a-a328b30441cc None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:51.910172 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f47742f-a91f-4762-8e9c-12e9683bc487 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:51.910947 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f47742f-a91f-4762-8e9c-12e9683bc487 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:51.911287 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f47742f-a91f-4762-8e9c-12e9683bc487 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:51.911626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f47742f-a91f-4762-8e9c-12e9683bc487 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:51.912265 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f47742f-a91f-4762-8e9c-12e9683bc487 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:51.912563 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f47742f-a91f-4762-8e9c-12e9683bc487 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:51.913912 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f47742f-a91f-4762-8e9c-12e9683bc487 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:51.926560 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 530/1067] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:51.932205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f47742f-a91f-4762-8e9c-12e9683bc487 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:51.956551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:51.989415 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2b73ea-0f27-4585-906a-a328b30441cc None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:52.024935 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d4c8bc-795d-45b4-a9a8-18b2f41ee0d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:52.025563 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d4c8bc-795d-45b4-a9a8-18b2f41ee0d8 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:52.025839 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d4c8bc-795d-45b4-a9a8-18b2f41ee0d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:52.026108 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d4c8bc-795d-45b4-a9a8-18b2f41ee0d8 None admin] PATH_INFO: `/v3/projects/5101d12502824233877ee3c3a7c29586` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:52.028182 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d4c8bc-795d-45b4-a9a8-18b2f41ee0d8 None admin] RBAC: Authorizing `identity:get_project(project_id=5101d12502824233877ee3c3a7c29586)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:52.029734 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d4c8bc-795d-45b4-a9a8-18b2f41ee0d8 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:52.031716 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 538/1068] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:48:51 2022] GET /identity/v3/projects/5101d12502824233877ee3c3a7c29586 => generated 311 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:52.037474 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d4c8bc-795d-45b4-a9a8-18b2f41ee0d8 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:52.040711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f47742f-a91f-4762-8e9c-12e9683bc487 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:52.063833 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:52.078257 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c6cf2d-66e1-474c-b630-e083ee8aac6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:52.078919 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c6cf2d-66e1-474c-b630-e083ee8aac6a None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:52.079556 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c6cf2d-66e1-474c-b630-e083ee8aac6a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:52.079906 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c6cf2d-66e1-474c-b630-e083ee8aac6a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:52.087106 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c6cf2d-66e1-474c-b630-e083ee8aac6a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:52.088613 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c6cf2d-66e1-474c-b630-e083ee8aac6a None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:52.101007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-76c6cf2d-66e1-474c-b630-e083ee8aac6a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:52.101007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:52.101007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:52.101007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:52.101007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:52.101007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:52.101007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:52.101007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:52.101007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:52.101007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:52.101007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:52.101007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:52.101007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:52.101007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:52.101007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:52.102101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:52.107970 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:52.107970 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:52.107970 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:52.107970 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:52.107970 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:52.107970 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:52.107970 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:52.107970 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:52.107970 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 531/1069] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:52.107970 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c6cf2d-66e1-474c-b630-e083ee8aac6a None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:52.146445 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d4c8bc-795d-45b4-a9a8-18b2f41ee0d8 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:52.182880 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74feea28-c70b-40fd-a365-bc2a9967b6ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:52.183526 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74feea28-c70b-40fd-a365-bc2a9967b6ea None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:52.183835 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74feea28-c70b-40fd-a365-bc2a9967b6ea None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:52.184108 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74feea28-c70b-40fd-a365-bc2a9967b6ea None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:52.184834 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74feea28-c70b-40fd-a365-bc2a9967b6ea None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=5101d12502824233877ee3c3a7c29586, user.id=b86b7649345244fd8fcd3a80e3024209) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:52.185138 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74feea28-c70b-40fd-a365-bc2a9967b6ea None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:52.186599 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74feea28-c70b-40fd-a365-bc2a9967b6ea None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:52.192129 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 539/1070] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:48:52 2022] GET /identity/v3/role_assignments?user.id=b86b7649345244fd8fcd3a80e3024209&scope.project.id=5101d12502824233877ee3c3a7c29586&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 259 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:52.197687 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:52.220707 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c6cf2d-66e1-474c-b630-e083ee8aac6a None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:52.266698 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e2a297-f3e8-4ed5-a94c-8e997123a8ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:52.267495 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e2a297-f3e8-4ed5-a94c-8e997123a8ae None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:52.267872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e2a297-f3e8-4ed5-a94c-8e997123a8ae None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:52.268240 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e2a297-f3e8-4ed5-a94c-8e997123a8ae None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:52.269050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e2a297-f3e8-4ed5-a94c-8e997123a8ae None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:52.269456 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e2a297-f3e8-4ed5-a94c-8e997123a8ae None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:52.271046 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e2a297-f3e8-4ed5-a94c-8e997123a8ae None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:52.286419 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 532/1071] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:52.292558 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74feea28-c70b-40fd-a365-bc2a9967b6ea None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:52.394741 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:52.408450 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74feea28-c70b-40fd-a365-bc2a9967b6ea None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:52.443303 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06377bdb-e6a1-457b-8b8c-06947d00873d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:52.443969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06377bdb-e6a1-457b-8b8c-06947d00873d None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:52.444309 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06377bdb-e6a1-457b-8b8c-06947d00873d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:52.444592 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06377bdb-e6a1-457b-8b8c-06947d00873d None admin] PATH_INFO: `/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/953cb6c24bcc4551b91ce540d18cd751/roles/d2787a643a3148c882b7a96a59e0a5e3` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:52.446755 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06377bdb-e6a1-457b-8b8c-06947d00873d None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4816e95144485e8135e5b8f68c3662, user_id=953cb6c24bcc4551b91ce540d18cd751, role_id=d2787a643a3148c882b7a96a59e0a5e3)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:52.448193 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06377bdb-e6a1-457b-8b8c-06947d00873d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:52.462198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 540/1072] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:48:52 2022] PUT /identity/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/953cb6c24bcc4551b91ce540d18cd751/roles/d2787a643a3148c882b7a96a59e0a5e3 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:48:53.601363 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a6a71a-060a-421e-98d8-c41c95ed1d46 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:53.601897 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a6a71a-060a-421e-98d8-c41c95ed1d46 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:53.601897 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a6a71a-060a-421e-98d8-c41c95ed1d46 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:53.603307 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 533/1073] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:53.609156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5722fb2-48e7-4f7f-b809-aefca945ed00 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:53.609596 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5722fb2-48e7-4f7f-b809-aefca945ed00 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:53.609743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5722fb2-48e7-4f7f-b809-aefca945ed00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:53.629075 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-c5722fb2-48e7-4f7f-b809-aefca945ed00 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:53.630275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-c5722fb2-48e7-4f7f-b809-aefca945ed00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:53.706142 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 541/1074] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:53.973053 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a6a71a-060a-421e-98d8-c41c95ed1d46 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:54.078281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a6a71a-060a-421e-98d8-c41c95ed1d46 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:54.112502 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e65e7d1-233e-47e6-a187-9aa4a3630d8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:54.113117 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e65e7d1-233e-47e6-a187-9aa4a3630d8d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:54.113397 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e65e7d1-233e-47e6-a187-9aa4a3630d8d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:54.113674 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e65e7d1-233e-47e6-a187-9aa4a3630d8d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:54.126127 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:54.182875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e65e7d1-233e-47e6-a187-9aa4a3630d8d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:54.184364 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e65e7d1-233e-47e6-a187-9aa4a3630d8d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:54.298159 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 534/1075] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:48:53 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:48:54.301645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5722fb2-48e7-4f7f-b809-aefca945ed00 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:54.370144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fe4cb5-1b79-4894-922a-55c84b55d1f2 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:54.370454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fe4cb5-1b79-4894-922a-55c84b55d1f2 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:54.370725 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fe4cb5-1b79-4894-922a-55c84b55d1f2 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:54.372706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 535/1076] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:48:54.377571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13152ef3-bec1-4816-a01a-b78b36573e7e None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:54.377860 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13152ef3-bec1-4816-a01a-b78b36573e7e None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:54.378171 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13152ef3-bec1-4816-a01a-b78b36573e7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:54.383698 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:54.396903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-13152ef3-bec1-4816-a01a-b78b36573e7e None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:54.398108 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-13152ef3-bec1-4816-a01a-b78b36573e7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:54.410232 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5722fb2-48e7-4f7f-b809-aefca945ed00 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:54.437219 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:54.449122 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb9f0ba-c639-4428-9047-a0c09024d4c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:54.449760 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb9f0ba-c639-4428-9047-a0c09024d4c3 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:54.450042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb9f0ba-c639-4428-9047-a0c09024d4c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:54.450399 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb9f0ba-c639-4428-9047-a0c09024d4c3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:54.460665 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb9f0ba-c639-4428-9047-a0c09024d4c3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:54.462138 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb9f0ba-c639-4428-9047-a0c09024d4c3 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:54.473429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 536/1077] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:48:54.478432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-3eb9f0ba-c639-4428-9047-a0c09024d4c3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:54.478432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:54.478432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:54.478432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:54.478432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:54.478432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:54.478432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:54.478432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:54.478432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:54.478432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:54.478432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:54.478432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:54.478432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:54.478432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:54.478432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:54.478432 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:54.479424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:54.480906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:54.480906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:54.480906 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:54.482457 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 542/1078] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:54.486619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13152ef3-bec1-4816-a01a-b78b36573e7e None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:54.593687 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13152ef3-bec1-4816-a01a-b78b36573e7e None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:54.609140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ede52f-70a7-4dd6-a5db-6c8868f01f6a None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:54.609429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ede52f-70a7-4dd6-a5db-6c8868f01f6a None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:54.609972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ede52f-70a7-4dd6-a5db-6c8868f01f6a None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:54.611095 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 543/1079] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:54.616027 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9520f2ca-d4bb-4103-8c74-2fd24d11ec71 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:54.616309 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9520f2ca-d4bb-4103-8c74-2fd24d11ec71 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:54.616582 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9520f2ca-d4bb-4103-8c74-2fd24d11ec71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:54.634615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-9520f2ca-d4bb-4103-8c74-2fd24d11ec71 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:54.635224 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de84991-51e2-4a80-ba92-b4328a383065 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:54.635813 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-9520f2ca-d4bb-4103-8c74-2fd24d11ec71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:54.635940 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de84991-51e2-4a80-ba92-b4328a383065 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:54.636099 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de84991-51e2-4a80-ba92-b4328a383065 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:54.636362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de84991-51e2-4a80-ba92-b4328a383065 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:54.637334 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de84991-51e2-4a80-ba92-b4328a383065 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:54.637676 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de84991-51e2-4a80-ba92-b4328a383065 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:54.639069 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de84991-51e2-4a80-ba92-b4328a383065 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:54.658138 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 537/1080] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:54.712534 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 544/1081] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:48:54.754404 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de84991-51e2-4a80-ba92-b4328a383065 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:54.789266 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:54.865784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de84991-51e2-4a80-ba92-b4328a383065 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:54.903692 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbe1c88-bf6d-4e16-ac12-c1066340ce49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:54.904325 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbe1c88-bf6d-4e16-ac12-c1066340ce49 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:54.904609 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbe1c88-bf6d-4e16-ac12-c1066340ce49 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:54.904882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbe1c88-bf6d-4e16-ac12-c1066340ce49 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:54.916511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebbe1c88-bf6d-4e16-ac12-c1066340ce49 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:54.921460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-ebbe1c88-bf6d-4e16-ac12-c1066340ce49 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:54.921460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:54.921460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:54.921460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:54.921460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:54.921460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:54.921460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:54.921460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:54.921460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:54.921460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:54.921460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:54.921460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:54.921460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:54.921460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:54.921460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:54.921460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:54.922549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:54.922549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:54.922549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:54.922549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:54.922549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:54.922549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:54.922549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:54.922549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:54.922549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:54.922549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:54.922549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:54.922549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:54.922549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:54.922549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:54.922549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:54.922549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:54.923573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:54.923573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:54.923573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:54.923573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:54.923573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:54.923573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:54.923573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:54.923573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:54.923573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:54.923573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:54.923573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:54.923573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:54.923573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 538/1082] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:48:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:54.929743 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9520f2ca-d4bb-4103-8c74-2fd24d11ec71 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:54.979421 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbe1c88-bf6d-4e16-ac12-c1066340ce49 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:54.995718 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:55.037163 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9520f2ca-d4bb-4103-8c74-2fd24d11ec71 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:55.071177 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:55.071473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36eafef9-f75e-4025-ad52-bdd3440be0dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:55.072084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eafef9-f75e-4025-ad52-bdd3440be0dd None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:55.072456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eafef9-f75e-4025-ad52-bdd3440be0dd None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:55.072653 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eafef9-f75e-4025-ad52-bdd3440be0dd None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:55.073438 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36eafef9-f75e-4025-ad52-bdd3440be0dd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:55.073751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36eafef9-f75e-4025-ad52-bdd3440be0dd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:55.075686 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36eafef9-f75e-4025-ad52-bdd3440be0dd None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:55.083352 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbe1c88-bf6d-4e16-ac12-c1066340ce49 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:55.093099 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 545/1083] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:48:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:55.094146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36eafef9-f75e-4025-ad52-bdd3440be0dd None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:55.117413 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a80dba5-9116-4709-b9d1-a7ac96ca75e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:55.117981 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a80dba5-9116-4709-b9d1-a7ac96ca75e8 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:55.118296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a80dba5-9116-4709-b9d1-a7ac96ca75e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:55.118597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a80dba5-9116-4709-b9d1-a7ac96ca75e8 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:55.129070 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a80dba5-9116-4709-b9d1-a7ac96ca75e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:55.133846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-3a80dba5-9116-4709-b9d1-a7ac96ca75e8 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 07 03:48:55.133846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:55.133846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:55.133846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:55.133846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:55.133846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:55.133846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:55.133846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:55.133846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:55.133846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:55.133846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:55.133846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:55.133846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:55.133846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:55.133846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:55.133846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:55.134903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:55.134903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:55.134903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:55.134903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:55.134903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:55.134903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:55.134903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:55.134903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:55.134903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:55.134903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:55.134903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:55.134903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:55.134903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:55.134903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:55.134903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:55.134903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:55.135810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:55.135810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:55.135810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:55.135810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:55.135810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:55.135810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:55.135810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:55.135810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:55.135810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:55.135810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:55.135810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 07 03:48:55.135810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:55.135810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 539/1084] 10.222.0.27 () {56 vars in 1120 bytes} [Wed Sep 7 03:48:54 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:55.141226 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a80dba5-9116-4709-b9d1-a7ac96ca75e8 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:55.193521 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36eafef9-f75e-4025-ad52-bdd3440be0dd None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:55.215485 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:55.227298 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a99af5-bc54-4a4e-930c-1712225cd21f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:55.227938 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a99af5-bc54-4a4e-930c-1712225cd21f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:55.228243 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a99af5-bc54-4a4e-930c-1712225cd21f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:55.228498 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a99af5-bc54-4a4e-930c-1712225cd21f None admin] PATH_INFO: `/v3/users/b86b7649345244fd8fcd3a80e3024209` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:55.229658 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a99af5-bc54-4a4e-930c-1712225cd21f None admin] RBAC: Authorizing `identity:get_user(user_id=b86b7649345244fd8fcd3a80e3024209)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:55.231719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a99af5-bc54-4a4e-930c-1712225cd21f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:55.233318 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 546/1085] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:48:55 2022] GET /identity/v3/users/b86b7649345244fd8fcd3a80e3024209 => generated 285 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:55.239048 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a99af5-bc54-4a4e-930c-1712225cd21f None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:55.240317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a80dba5-9116-4709-b9d1-a7ac96ca75e8 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:55.276351 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba649b9b-4a5c-47ca-a8f6-121dd62349f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:55.276964 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba649b9b-4a5c-47ca-a8f6-121dd62349f0 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:55.277808 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba649b9b-4a5c-47ca-a8f6-121dd62349f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:55.278089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba649b9b-4a5c-47ca-a8f6-121dd62349f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:55.278786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba649b9b-4a5c-47ca-a8f6-121dd62349f0 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:55.279094 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba649b9b-4a5c-47ca-a8f6-121dd62349f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:55.280459 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba649b9b-4a5c-47ca-a8f6-121dd62349f0 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:55.288283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 540/1086] 10.222.0.27 () {56 vars in 1115 bytes} [Wed Sep 7 03:48:55 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:55.293962 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba649b9b-4a5c-47ca-a8f6-121dd62349f0 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:55.309027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:55.345581 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a99af5-bc54-4a4e-930c-1712225cd21f None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:55.379069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acc3a70-c3ec-4a08-bbdd-5291af4d5665 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:55.379630 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acc3a70-c3ec-4a08-bbdd-5291af4d5665 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:55.379911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acc3a70-c3ec-4a08-bbdd-5291af4d5665 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:55.380185 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acc3a70-c3ec-4a08-bbdd-5291af4d5665 None admin] PATH_INFO: `/v3/projects/5101d12502824233877ee3c3a7c29586` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:55.381305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acc3a70-c3ec-4a08-bbdd-5291af4d5665 None admin] RBAC: Authorizing `identity:get_project(project_id=5101d12502824233877ee3c3a7c29586)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:55.382774 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acc3a70-c3ec-4a08-bbdd-5291af4d5665 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:55.384744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 547/1087] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:48:55 2022] GET /identity/v3/projects/5101d12502824233877ee3c3a7c29586 => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:55.390713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acc3a70-c3ec-4a08-bbdd-5291af4d5665 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:55.399033 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba649b9b-4a5c-47ca-a8f6-121dd62349f0 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:55.433282 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7054b4-69ef-436b-9a40-1af970c0f33f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:55.434510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7054b4-69ef-436b-9a40-1af970c0f33f None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:55.434751 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7054b4-69ef-436b-9a40-1af970c0f33f None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:55.435067 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7054b4-69ef-436b-9a40-1af970c0f33f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:55.445168 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7054b4-69ef-436b-9a40-1af970c0f33f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:55.446628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7054b4-69ef-436b-9a40-1af970c0f33f None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:55.448114 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:55.461763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-ea7054b4-69ef-436b-9a40-1af970c0f33f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:55.461763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:55.461763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:55.461763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:55.461763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:55.461763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:55.461763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:55.461763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:55.461763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:55.461763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:55.461763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:55.461763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:55.461763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:55.461763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:55.461763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:55.461763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:55.462938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:55.463862 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:55.463862 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:55.463862 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:55.463862 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 541/1088] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:55.469278 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7054b4-69ef-436b-9a40-1af970c0f33f None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:55.490850 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acc3a70-c3ec-4a08-bbdd-5291af4d5665 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:55.523627 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c015a1c2-a189-443f-a40a-245dabc5edea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:55.524285 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c015a1c2-a189-443f-a40a-245dabc5edea None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:55.524567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c015a1c2-a189-443f-a40a-245dabc5edea None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:55.524842 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c015a1c2-a189-443f-a40a-245dabc5edea None admin] PATH_INFO: `/v3/projects/5101d12502824233877ee3c3a7c29586/users/b86b7649345244fd8fcd3a80e3024209/roles/9d5b19a61c46475d8048b1bd66ca25a1` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:55.526826 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c015a1c2-a189-443f-a40a-245dabc5edea None admin] RBAC: Authorizing `identity:create_grant(project_id=5101d12502824233877ee3c3a7c29586, user_id=b86b7649345244fd8fcd3a80e3024209, role_id=9d5b19a61c46475d8048b1bd66ca25a1)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:55.528860 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c015a1c2-a189-443f-a40a-245dabc5edea None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:55.532895 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:55.542041 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 548/1089] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:48:55 2022] PUT /identity/v3/projects/5101d12502824233877ee3c3a7c29586/users/b86b7649345244fd8fcd3a80e3024209/roles/9d5b19a61c46475d8048b1bd66ca25a1 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 07 03:48:55.572086 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7054b4-69ef-436b-9a40-1af970c0f33f None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:55.610006 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7ae556-5fcb-456f-8876-b10e963ee903 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:55.610656 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7ae556-5fcb-456f-8876-b10e963ee903 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:55.610934 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7ae556-5fcb-456f-8876-b10e963ee903 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:55.611208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7ae556-5fcb-456f-8876-b10e963ee903 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:55.611873 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7ae556-5fcb-456f-8876-b10e963ee903 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:55.612283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7ae556-5fcb-456f-8876-b10e963ee903 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:55.613722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7ae556-5fcb-456f-8876-b10e963ee903 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:55.627627 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 542/1090] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:48:55.633656 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c015a1c2-a189-443f-a40a-245dabc5edea None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:55.740797 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c015a1c2-a189-443f-a40a-245dabc5edea None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:55.757299 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:55.774981 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777db914-bffe-4b53-abf3-16619b8a824e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:55.775611 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777db914-bffe-4b53-abf3-16619b8a824e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:55.775892 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777db914-bffe-4b53-abf3-16619b8a824e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:55.776167 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777db914-bffe-4b53-abf3-16619b8a824e None admin] PATH_INFO: `/v3/users/nova` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:55.784161 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777db914-bffe-4b53-abf3-16619b8a824e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:55.785605 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777db914-bffe-4b53-abf3-16619b8a824e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:55.796781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-777db914-bffe-4b53-abf3-16619b8a824e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:48:55.796781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:55.796781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:55.796781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:55.796781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:55.796781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:55.796781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:55.796781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:55.796781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:55.796781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:55.796781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:55.796781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:55.796781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:48:55.796781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:48:55.796781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:48:55.796781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:55.797751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:55.799200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:55.799200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:55.799200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:48:55.799200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:48:55.799200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:48:55.799200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:48:55.799200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:48:55.799200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:48:55.799200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 07 03:48:55.799200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:55.799200 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 549/1091] 10.222.0.27 () {56 vars in 1093 bytes} [Wed Sep 7 03:48:55 2022] GET /identity/v3/users/nova => generated 82 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:55.803761 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7ae556-5fcb-456f-8876-b10e963ee903 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:55.902170 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7ae556-5fcb-456f-8876-b10e963ee903 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:55.927310 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:55.936231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77908180-b2e6-4144-bfd8-8f9fc094a092 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:55.936833 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77908180-b2e6-4144-bfd8-8f9fc094a092 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:55.937110 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77908180-b2e6-4144-bfd8-8f9fc094a092 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:55.937380 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77908180-b2e6-4144-bfd8-8f9fc094a092 None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:55.938106 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77908180-b2e6-4144-bfd8-8f9fc094a092 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:55.938435 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77908180-b2e6-4144-bfd8-8f9fc094a092 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:55.939841 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77908180-b2e6-4144-bfd8-8f9fc094a092 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:55.957210 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 543/1092] 10.222.0.27 () {56 vars in 1133 bytes} [Wed Sep 7 03:48:55 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:55.962899 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777db914-bffe-4b53-abf3-16619b8a824e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:56.065810 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777db914-bffe-4b53-abf3-16619b8a824e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:56.106028 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8620d0c0-7663-4716-a374-5fbe51dda0f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:56.106669 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8620d0c0-7663-4716-a374-5fbe51dda0f3 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:56.106951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8620d0c0-7663-4716-a374-5fbe51dda0f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:56.107227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8620d0c0-7663-4716-a374-5fbe51dda0f3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:56.151777 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:56.173286 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8620d0c0-7663-4716-a374-5fbe51dda0f3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:56.174690 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8620d0c0-7663-4716-a374-5fbe51dda0f3 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:56.272584 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 550/1093] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:48:55 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:48:56.278397 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77908180-b2e6-4144-bfd8-8f9fc094a092 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:56.376782 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77908180-b2e6-4144-bfd8-8f9fc094a092 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:56.411105 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568feb31-16d7-4356-9ef4-906953eceb3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:56.411757 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568feb31-16d7-4356-9ef4-906953eceb3e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:56.412038 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568feb31-16d7-4356-9ef4-906953eceb3e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:56.412313 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568feb31-16d7-4356-9ef4-906953eceb3e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:56.423360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568feb31-16d7-4356-9ef4-906953eceb3e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:56.424739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568feb31-16d7-4356-9ef4-906953eceb3e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:56.439922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-568feb31-16d7-4356-9ef4-906953eceb3e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:56.439922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:56.439922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:56.439922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:56.439922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:56.439922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:56.439922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:56.439922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:56.439922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:56.439922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:56.439922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:56.439922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:56.439922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:48:56.439922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:48:56.439922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:48:56.439922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:56.441017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:48:56.441923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:48:56.441923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:48:56.441923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:56.441923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 544/1094] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:48:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:48:56.446859 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8620d0c0-7663-4716-a374-5fbe51dda0f3 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:56.522822 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:56.545044 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8620d0c0-7663-4716-a374-5fbe51dda0f3 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:56.579699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3735130-55a9-441f-8fb5-e27eca0c4075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:56.580298 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3735130-55a9-441f-8fb5-e27eca0c4075 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:56.580576 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3735130-55a9-441f-8fb5-e27eca0c4075 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:56.580850 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3735130-55a9-441f-8fb5-e27eca0c4075 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:56.581509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3735130-55a9-441f-8fb5-e27eca0c4075 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:56.581811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3735130-55a9-441f-8fb5-e27eca0c4075 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:56.583210 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3735130-55a9-441f-8fb5-e27eca0c4075 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:56.594360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 551/1095] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:48:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:56.599963 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568feb31-16d7-4356-9ef4-906953eceb3e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:56.636025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:56.699920 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568feb31-16d7-4356-9ef4-906953eceb3e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:56.733536 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1037d2e1-b561-4d09-bde3-410ebed3135e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:56.734186 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1037d2e1-b561-4d09-bde3-410ebed3135e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:56.734467 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1037d2e1-b561-4d09-bde3-410ebed3135e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:56.734740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1037d2e1-b561-4d09-bde3-410ebed3135e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:56.741374 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1037d2e1-b561-4d09-bde3-410ebed3135e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:56.742848 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1037d2e1-b561-4d09-bde3-410ebed3135e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:56.754482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-1037d2e1-b561-4d09-bde3-410ebed3135e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:56.754482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:56.754482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:56.754482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:56.754482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:56.754482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:56.754482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:56.754482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:56.754482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:56.754482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:56.754482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:56.754482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:56.754482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:48:56.754482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:48:56.754482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:56.755579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:56.756941 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:48:56.756941 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:48:56.756941 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:48:56.756941 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:48:56.756941 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:48:56.756941 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:48:56.756941 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:48:56.756941 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:48:56.756941 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 545/1096] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:48:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:56.761358 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3735130-55a9-441f-8fb5-e27eca0c4075 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:56.860709 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3735130-55a9-441f-8fb5-e27eca0c4075 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:56.893458 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e06928-fa85-43ad-806d-ef51cfeedc56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:56.894071 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e06928-fa85-43ad-806d-ef51cfeedc56 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:56.894379 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e06928-fa85-43ad-806d-ef51cfeedc56 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:56.894652 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e06928-fa85-43ad-806d-ef51cfeedc56 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:56.895339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e06928-fa85-43ad-806d-ef51cfeedc56 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:56.895644 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e06928-fa85-43ad-806d-ef51cfeedc56 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:56.897073 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e06928-fa85-43ad-806d-ef51cfeedc56 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:56.904720 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:56.910770 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 552/1097] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:48:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:56.917009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1037d2e1-b561-4d09-bde3-410ebed3135e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:57.008587 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:57.014755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1037d2e1-b561-4d09-bde3-410ebed3135e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:57.048151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5f6398-ad0c-496c-a366-a6121b87526a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:57.048748 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5f6398-ad0c-496c-a366-a6121b87526a None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:57.049030 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5f6398-ad0c-496c-a366-a6121b87526a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:57.049305 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5f6398-ad0c-496c-a366-a6121b87526a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:57.050027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5f6398-ad0c-496c-a366-a6121b87526a None admin] RBAC: Adding query filter params (role.id=d2787a643a3148c882b7a96a59e0a5e3, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=953cb6c24bcc4551b91ce540d18cd751) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:57.050368 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5f6398-ad0c-496c-a366-a6121b87526a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:57.051788 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5f6398-ad0c-496c-a366-a6121b87526a None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:57.057482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 546/1098] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:48:56 2022] GET /identity/v3/role_assignments?user.id=953cb6c24bcc4551b91ce540d18cd751&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=d2787a643a3148c882b7a96a59e0a5e3 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:57.695211 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8962ddd-9bf0-4567-8b89-ebb19afe92b6 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:57.695498 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8962ddd-9bf0-4567-8b89-ebb19afe92b6 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:57.695778 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8962ddd-9bf0-4567-8b89-ebb19afe92b6 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:57.697150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 553/1099] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:48:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:48:57.702638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2529cd3c-d1ed-412b-9246-15f401aab6c1 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:57.702925 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2529cd3c-d1ed-412b-9246-15f401aab6c1 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:57.703201 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2529cd3c-d1ed-412b-9246-15f401aab6c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:57.723301 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-2529cd3c-d1ed-412b-9246-15f401aab6c1 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:48:57.724489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-2529cd3c-d1ed-412b-9246-15f401aab6c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:48:57.793335 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 547/1100] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:48:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:48:58.070973 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8962ddd-9bf0-4567-8b89-ebb19afe92b6 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:58.170908 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8962ddd-9bf0-4567-8b89-ebb19afe92b6 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:58.205056 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0c0556-801a-4b06-b428-b42227580795 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:58.205678 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0c0556-801a-4b06-b428-b42227580795 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:58.205958 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0c0556-801a-4b06-b428-b42227580795 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:58.206264 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0c0556-801a-4b06-b428-b42227580795 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:58.216507 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0c0556-801a-4b06-b428-b42227580795 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:48:58.221308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-0a0c0556-801a-4b06-b428-b42227580795 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:58.221308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:48:58.221308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:48:58.221308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:48:58.221308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:48:58.221308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:48:58.221308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:48:58.221308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:48:58.221308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:48:58.221308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:48:58.221308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:48:58.221308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:48:58.221308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:48:58.221308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:48:58.221308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:48:58.221308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:48:58.222463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:48:58.222463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:48:58.222463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:48:58.222463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:48:58.222463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:48:58.222463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:48:58.222463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:48:58.222463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:48:58.222463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:48:58.222463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:48:58.222463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:48:58.222463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:48:58.222463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:48:58.222463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:48:58.222463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:48:58.222463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:48:58.223751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:48:58.223751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:48:58.223751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:48:58.223751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:48:58.223751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:48:58.223751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:48:58.223751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:48:58.223751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:48:58.223751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:48:58.223751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:48:58.223751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:48:58.223751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:48:58.223751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 554/1101] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:48:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:48:58.228384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2529cd3c-d1ed-412b-9246-15f401aab6c1 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:58.315418 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:58.325118 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2529cd3c-d1ed-412b-9246-15f401aab6c1 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:58.359166 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8995fb6a-2100-4160-9eff-a6e5210781b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:58.359774 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8995fb6a-2100-4160-9eff-a6e5210781b3 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:58.360057 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8995fb6a-2100-4160-9eff-a6e5210781b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:58.360330 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8995fb6a-2100-4160-9eff-a6e5210781b3 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:58.360990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8995fb6a-2100-4160-9eff-a6e5210781b3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:58.361291 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8995fb6a-2100-4160-9eff-a6e5210781b3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:58.362681 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8995fb6a-2100-4160-9eff-a6e5210781b3 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:58.370497 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 548/1102] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:48:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:58.377098 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0c0556-801a-4b06-b428-b42227580795 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:58.434784 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:58.475578 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0c0556-801a-4b06-b428-b42227580795 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:58.513321 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55579e93-76cc-495e-863b-660567edc0a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:58.514234 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55579e93-76cc-495e-863b-660567edc0a0 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:58.514652 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55579e93-76cc-495e-863b-660567edc0a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:58.515060 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55579e93-76cc-495e-863b-660567edc0a0 None admin] PATH_INFO: `/v3/users/b86b7649345244fd8fcd3a80e3024209` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:58.516173 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55579e93-76cc-495e-863b-660567edc0a0 None admin] RBAC: Authorizing `identity:get_user(user_id=b86b7649345244fd8fcd3a80e3024209)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:58.517593 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55579e93-76cc-495e-863b-660567edc0a0 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:58.519193 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 555/1103] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:48:58 2022] GET /identity/v3/users/b86b7649345244fd8fcd3a80e3024209 => generated 285 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:58.524983 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8995fb6a-2100-4160-9eff-a6e5210781b3 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:58.621608 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8995fb6a-2100-4160-9eff-a6e5210781b3 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:58.654340 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff08dd84-a39a-4ce4-bffb-50303835013d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:58.654931 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff08dd84-a39a-4ce4-bffb-50303835013d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:58.655212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff08dd84-a39a-4ce4-bffb-50303835013d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:58.655485 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff08dd84-a39a-4ce4-bffb-50303835013d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:58.675634 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:58.720634 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff08dd84-a39a-4ce4-bffb-50303835013d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:58.722053 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff08dd84-a39a-4ce4-bffb-50303835013d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:58.818271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 549/1104] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:48:58 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:48:58.824344 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55579e93-76cc-495e-863b-660567edc0a0 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:58.920352 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55579e93-76cc-495e-863b-660567edc0a0 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:58.945409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:58.954601 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a7198a-00da-48a8-8adc-4e51ee0c26d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:58.955225 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a7198a-00da-48a8-8adc-4e51ee0c26d2 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:58.955505 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a7198a-00da-48a8-8adc-4e51ee0c26d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:58.955777 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a7198a-00da-48a8-8adc-4e51ee0c26d2 None admin] PATH_INFO: `/v3/projects/5101d12502824233877ee3c3a7c29586` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:58.956891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a7198a-00da-48a8-8adc-4e51ee0c26d2 None admin] RBAC: Authorizing `identity:get_project(project_id=5101d12502824233877ee3c3a7c29586)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:58.958346 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a7198a-00da-48a8-8adc-4e51ee0c26d2 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:58.959964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 556/1105] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:48:58 2022] GET /identity/v3/projects/5101d12502824233877ee3c3a7c29586 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:48:58.966077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff08dd84-a39a-4ce4-bffb-50303835013d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:48:59.016846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:48:59.063662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff08dd84-a39a-4ce4-bffb-50303835013d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:48:59.097268 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05c42e5-cc29-44ae-9d7c-8f2485b78d0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:48:59.097851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05c42e5-cc29-44ae-9d7c-8f2485b78d0d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:48:59.098145 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05c42e5-cc29-44ae-9d7c-8f2485b78d0d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:48:59.098421 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05c42e5-cc29-44ae-9d7c-8f2485b78d0d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:48:59.099126 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05c42e5-cc29-44ae-9d7c-8f2485b78d0d None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=5101d12502824233877ee3c3a7c29586, user.id=b86b7649345244fd8fcd3a80e3024209) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:48:59.099426 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05c42e5-cc29-44ae-9d7c-8f2485b78d0d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:48:59.100822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05c42e5-cc29-44ae-9d7c-8f2485b78d0d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:48:59.106597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 550/1106] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:48:58 2022] GET /identity/v3/role_assignments?user.id=b86b7649345244fd8fcd3a80e3024209&scope.project.id=5101d12502824233877ee3c3a7c29586&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:01.255372 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c993be3d-10ab-4cf5-840f-bb928119a568 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:01.255893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c993be3d-10ab-4cf5-840f-bb928119a568 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:01.256007 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c993be3d-10ab-4cf5-840f-bb928119a568 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:01.257259 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 557/1107] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:49:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:49:01.263228 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eea8ef-9baf-402f-8df4-aabb3d89b4ce None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:01.263625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eea8ef-9baf-402f-8df4-aabb3d89b4ce None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:01.263846 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eea8ef-9baf-402f-8df4-aabb3d89b4ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:01.282217 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-73eea8ef-9baf-402f-8df4-aabb3d89b4ce None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:49:01.283417 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-73eea8ef-9baf-402f-8df4-aabb3d89b4ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:49:01.348969 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 551/1108] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:49:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:49:01.619163 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c993be3d-10ab-4cf5-840f-bb928119a568 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:01.718790 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c993be3d-10ab-4cf5-840f-bb928119a568 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:01.752341 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1c802e-7b52-4163-8e20-26a0aa734abd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:01.752958 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1c802e-7b52-4163-8e20-26a0aa734abd None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:01.753237 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1c802e-7b52-4163-8e20-26a0aa734abd None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:01.753508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1c802e-7b52-4163-8e20-26a0aa734abd None admin] PATH_INFO: `/v3/domains/default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:01.754623 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1c802e-7b52-4163-8e20-26a0aa734abd None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:01.756047 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1c802e-7b52-4163-8e20-26a0aa734abd None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:01.757590 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 558/1109] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:49:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:01.763927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73eea8ef-9baf-402f-8df4-aabb3d89b4ce None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:01.808300 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:01.859515 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73eea8ef-9baf-402f-8df4-aabb3d89b4ce None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:01.893789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a434ad57-9317-4dfc-b7b1-58a08978e476 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:01.894561 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a434ad57-9317-4dfc-b7b1-58a08978e476 None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:01.894972 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a434ad57-9317-4dfc-b7b1-58a08978e476 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:01.895341 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a434ad57-9317-4dfc-b7b1-58a08978e476 None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:01.896250 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a434ad57-9317-4dfc-b7b1-58a08978e476 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:01.898103 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a434ad57-9317-4dfc-b7b1-58a08978e476 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:01.920196 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 552/1110] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:49:01 2022] POST /identity/v3/users => generated 286 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:49:04.086556 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3490dc43-e1b8-44e4-8a48-375936037ac2 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:04.087089 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3490dc43-e1b8-44e4-8a48-375936037ac2 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:04.087089 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3490dc43-e1b8-44e4-8a48-375936037ac2 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:04.088435 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 559/1111] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:49:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:49:04.093768 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a99ceff-8c2a-4f78-a9f8-e07732e26b19 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:04.094155 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a99ceff-8c2a-4f78-a9f8-e07732e26b19 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:04.094352 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a99ceff-8c2a-4f78-a9f8-e07732e26b19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:04.112762 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-5a99ceff-8c2a-4f78-a9f8-e07732e26b19 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:49:04.113951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-5a99ceff-8c2a-4f78-a9f8-e07732e26b19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:49:04.178176 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 553/1112] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:49:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:49:04.445496 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3490dc43-e1b8-44e4-8a48-375936037ac2 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:04.507470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:04.546001 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3490dc43-e1b8-44e4-8a48-375936037ac2 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:04.580732 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7771a18-61a4-4323-ab2a-52187c93b05e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:04.581387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7771a18-61a4-4323-ab2a-52187c93b05e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:04.581666 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7771a18-61a4-4323-ab2a-52187c93b05e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:04.581940 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7771a18-61a4-4323-ab2a-52187c93b05e None admin] PATH_INFO: `/v3/roles/285107fb9b2449fe9f1d29f3075a246f` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:04.583380 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7771a18-61a4-4323-ab2a-52187c93b05e None admin] RBAC: Authorizing `identity:get_role(role_id=285107fb9b2449fe9f1d29f3075a246f)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:04.584744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7771a18-61a4-4323-ab2a-52187c93b05e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:04.586107 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 560/1113] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:49:04 2022] GET /identity/v3/roles/285107fb9b2449fe9f1d29f3075a246f => generated 223 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:04.592305 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a99ceff-8c2a-4f78-a9f8-e07732e26b19 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:04.622359 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:04.689851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a99ceff-8c2a-4f78-a9f8-e07732e26b19 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:04.723163 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383f2de5-c74d-4057-bbc7-df2a958e3d75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:04.723733 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383f2de5-c74d-4057-bbc7-df2a958e3d75 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:04.724010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383f2de5-c74d-4057-bbc7-df2a958e3d75 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:04.724284 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383f2de5-c74d-4057-bbc7-df2a958e3d75 None admin] PATH_INFO: `/v3/users/e9b5543320c942479f72653554e862e8` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:04.737417 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383f2de5-c74d-4057-bbc7-df2a958e3d75 None admin] RBAC: Authorizing `identity:get_user(user_id=e9b5543320c942479f72653554e862e8)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:04.738899 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383f2de5-c74d-4057-bbc7-df2a958e3d75 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:04.740481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 554/1114] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:49:04 2022] GET /identity/v3/users/e9b5543320c942479f72653554e862e8 => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:04.747005 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7771a18-61a4-4323-ab2a-52187c93b05e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:04.844534 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7771a18-61a4-4323-ab2a-52187c93b05e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:04.872899 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:04.878998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370bcb07-a4d2-4f7d-87a8-6b9c0581a655 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:04.879602 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370bcb07-a4d2-4f7d-87a8-6b9c0581a655 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:04.879884 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370bcb07-a4d2-4f7d-87a8-6b9c0581a655 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:04.880160 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370bcb07-a4d2-4f7d-87a8-6b9c0581a655 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:04.945364 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370bcb07-a4d2-4f7d-87a8-6b9c0581a655 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:04.946852 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370bcb07-a4d2-4f7d-87a8-6b9c0581a655 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:05.041397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 561/1115] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:49:04 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:49:05.047772 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383f2de5-c74d-4057-bbc7-df2a958e3d75 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:05.137521 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:05.147249 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383f2de5-c74d-4057-bbc7-df2a958e3d75 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:05.181054 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5068c5b5-e862-45d3-be60-f1aadc3eb8e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:05.181675 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5068c5b5-e862-45d3-be60-f1aadc3eb8e0 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:05.181958 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5068c5b5-e862-45d3-be60-f1aadc3eb8e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:05.182266 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5068c5b5-e862-45d3-be60-f1aadc3eb8e0 None admin] PATH_INFO: `/v3/projects/5101d12502824233877ee3c3a7c29586` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:05.183525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5068c5b5-e862-45d3-be60-f1aadc3eb8e0 None admin] RBAC: Authorizing `identity:get_project(project_id=5101d12502824233877ee3c3a7c29586)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:05.184940 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5068c5b5-e862-45d3-be60-f1aadc3eb8e0 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:05.186586 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 555/1116] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:49:05 2022] GET /identity/v3/projects/5101d12502824233877ee3c3a7c29586 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:05.192676 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370bcb07-a4d2-4f7d-87a8-6b9c0581a655 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:05.222178 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:05.289779 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370bcb07-a4d2-4f7d-87a8-6b9c0581a655 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:05.322697 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2c6d93-4aa8-42c9-917c-bd31d4f66a4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:05.323240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2c6d93-4aa8-42c9-917c-bd31d4f66a4e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:05.323519 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2c6d93-4aa8-42c9-917c-bd31d4f66a4e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:05.323793 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2c6d93-4aa8-42c9-917c-bd31d4f66a4e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:05.324508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2c6d93-4aa8-42c9-917c-bd31d4f66a4e None admin] RBAC: Adding query filter params (role.id=285107fb9b2449fe9f1d29f3075a246f, scope.project.id=5101d12502824233877ee3c3a7c29586, user.id=e9b5543320c942479f72653554e862e8) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:49:05.324815 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2c6d93-4aa8-42c9-917c-bd31d4f66a4e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:05.326306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2c6d93-4aa8-42c9-917c-bd31d4f66a4e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:05.331786 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 562/1117] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:49:05 2022] GET /identity/v3/role_assignments?user.id=e9b5543320c942479f72653554e862e8&scope.project.id=5101d12502824233877ee3c3a7c29586&role.id=285107fb9b2449fe9f1d29f3075a246f => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:07.459192 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b0931e-7853-4547-ac22-99c6670383da None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:07.459718 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b0931e-7853-4547-ac22-99c6670383da None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:07.459718 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b0931e-7853-4547-ac22-99c6670383da None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:07.461061 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 556/1118] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:49:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:49:07.466991 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df36885-ebfa-4a3d-85b8-9b476dc6c57e None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:07.467391 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df36885-ebfa-4a3d-85b8-9b476dc6c57e None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:07.467550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df36885-ebfa-4a3d-85b8-9b476dc6c57e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:07.485960 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-0df36885-ebfa-4a3d-85b8-9b476dc6c57e None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:49:07.487187 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-0df36885-ebfa-4a3d-85b8-9b476dc6c57e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:49:07.551523 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 563/1119] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:49:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:49:07.823359 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b0931e-7853-4547-ac22-99c6670383da None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:07.921515 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b0931e-7853-4547-ac22-99c6670383da None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:07.956460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e2fc3e-8fe5-4a9f-b83f-77935bb06e9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:07.957076 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e2fc3e-8fe5-4a9f-b83f-77935bb06e9d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:07.957354 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e2fc3e-8fe5-4a9f-b83f-77935bb06e9d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:07.957624 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e2fc3e-8fe5-4a9f-b83f-77935bb06e9d None admin] PATH_INFO: `/v3/roles/285107fb9b2449fe9f1d29f3075a246f` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:07.959021 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e2fc3e-8fe5-4a9f-b83f-77935bb06e9d None admin] RBAC: Authorizing `identity:get_role(role_id=285107fb9b2449fe9f1d29f3075a246f)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:07.960368 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e2fc3e-8fe5-4a9f-b83f-77935bb06e9d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:07.961672 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 557/1120] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:49:07 2022] GET /identity/v3/roles/285107fb9b2449fe9f1d29f3075a246f => generated 223 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:07.968093 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df36885-ebfa-4a3d-85b8-9b476dc6c57e None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:07.992048 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:08.065595 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df36885-ebfa-4a3d-85b8-9b476dc6c57e None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:08.099350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a5019a-d2da-4b68-b30f-7f8ed66aa51c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:08.099913 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a5019a-d2da-4b68-b30f-7f8ed66aa51c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:08.100192 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a5019a-d2da-4b68-b30f-7f8ed66aa51c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:08.100464 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a5019a-d2da-4b68-b30f-7f8ed66aa51c None admin] PATH_INFO: `/v3/users/e9b5543320c942479f72653554e862e8` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:08.101567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a5019a-d2da-4b68-b30f-7f8ed66aa51c None admin] RBAC: Authorizing `identity:get_user(user_id=e9b5543320c942479f72653554e862e8)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:08.103033 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a5019a-d2da-4b68-b30f-7f8ed66aa51c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:08.104602 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 564/1121] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:49:07 2022] GET /identity/v3/users/e9b5543320c942479f72653554e862e8 => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:08.110341 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e2fc3e-8fe5-4a9f-b83f-77935bb06e9d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:08.137958 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:08.209715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e2fc3e-8fe5-4a9f-b83f-77935bb06e9d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:08.241950 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640292e0-93fb-481c-a3e0-c14c5e0c22ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:08.242606 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640292e0-93fb-481c-a3e0-c14c5e0c22ea None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:08.242884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640292e0-93fb-481c-a3e0-c14c5e0c22ea None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:08.243158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640292e0-93fb-481c-a3e0-c14c5e0c22ea None admin] PATH_INFO: `/v3/projects/5101d12502824233877ee3c3a7c29586` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:08.244279 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640292e0-93fb-481c-a3e0-c14c5e0c22ea None admin] RBAC: Authorizing `identity:get_project(project_id=5101d12502824233877ee3c3a7c29586)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:08.246394 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640292e0-93fb-481c-a3e0-c14c5e0c22ea None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:08.248014 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 558/1122] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:49:08 2022] GET /identity/v3/projects/5101d12502824233877ee3c3a7c29586 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:08.253831 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a5019a-d2da-4b68-b30f-7f8ed66aa51c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:08.342566 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:08.351285 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a5019a-d2da-4b68-b30f-7f8ed66aa51c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:08.384139 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19323f86-6a37-43b4-8eb5-2c5cb8b7f014 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:08.384792 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19323f86-6a37-43b4-8eb5-2c5cb8b7f014 None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:08.385072 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19323f86-6a37-43b4-8eb5-2c5cb8b7f014 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:08.385344 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19323f86-6a37-43b4-8eb5-2c5cb8b7f014 None admin] PATH_INFO: `/v3/projects/5101d12502824233877ee3c3a7c29586/users/e9b5543320c942479f72653554e862e8/roles/285107fb9b2449fe9f1d29f3075a246f` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:08.388210 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19323f86-6a37-43b4-8eb5-2c5cb8b7f014 None admin] RBAC: Authorizing `identity:create_grant(project_id=5101d12502824233877ee3c3a7c29586, user_id=e9b5543320c942479f72653554e862e8, role_id=285107fb9b2449fe9f1d29f3075a246f)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:08.389641 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19323f86-6a37-43b4-8eb5-2c5cb8b7f014 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:08.401517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 565/1123] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:49:08 2022] PUT /identity/v3/projects/5101d12502824233877ee3c3a7c29586/users/e9b5543320c942479f72653554e862e8/roles/285107fb9b2449fe9f1d29f3075a246f => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:49:10.509167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ba6a0b-f17a-4b59-90be-14603c3a7e90 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:10.509691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ba6a0b-f17a-4b59-90be-14603c3a7e90 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:10.509691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ba6a0b-f17a-4b59-90be-14603c3a7e90 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:10.511025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 559/1124] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:49:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:49:10.516161 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45099da3-57f9-4b38-addf-92139f93a388 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:10.516586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45099da3-57f9-4b38-addf-92139f93a388 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:10.516704 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45099da3-57f9-4b38-addf-92139f93a388 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:10.535165 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-45099da3-57f9-4b38-addf-92139f93a388 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:49:10.536333 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-45099da3-57f9-4b38-addf-92139f93a388 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:49:10.601153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 566/1125] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:49:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:49:10.872940 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ba6a0b-f17a-4b59-90be-14603c3a7e90 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:10.965430 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:10.973238 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ba6a0b-f17a-4b59-90be-14603c3a7e90 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:11.005300 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0499f0d4-039f-4627-b24a-ccc3c3fefa6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:11.005915 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0499f0d4-039f-4627-b24a-ccc3c3fefa6e None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:11.006217 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0499f0d4-039f-4627-b24a-ccc3c3fefa6e None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:11.006492 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0499f0d4-039f-4627-b24a-ccc3c3fefa6e None admin] PATH_INFO: `/v3/roles/285107fb9b2449fe9f1d29f3075a246f` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:11.007871 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0499f0d4-039f-4627-b24a-ccc3c3fefa6e None admin] RBAC: Authorizing `identity:get_role(role_id=285107fb9b2449fe9f1d29f3075a246f)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:11.009217 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0499f0d4-039f-4627-b24a-ccc3c3fefa6e None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:11.010550 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 560/1126] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:49:10 2022] GET /identity/v3/roles/285107fb9b2449fe9f1d29f3075a246f => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:11.016803 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45099da3-57f9-4b38-addf-92139f93a388 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:11.095540 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:11.113689 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45099da3-57f9-4b38-addf-92139f93a388 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:11.147205 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6361e4a7-78c5-4bb4-8419-3000fea6ddde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:11.147758 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6361e4a7-78c5-4bb4-8419-3000fea6ddde None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:11.148041 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6361e4a7-78c5-4bb4-8419-3000fea6ddde None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:11.148319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6361e4a7-78c5-4bb4-8419-3000fea6ddde None admin] PATH_INFO: `/v3/users/e9b5543320c942479f72653554e862e8` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:11.149440 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6361e4a7-78c5-4bb4-8419-3000fea6ddde None admin] RBAC: Authorizing `identity:get_user(user_id=e9b5543320c942479f72653554e862e8)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:11.150907 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6361e4a7-78c5-4bb4-8419-3000fea6ddde None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:11.152741 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 567/1127] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:49:11 2022] GET /identity/v3/users/e9b5543320c942479f72653554e862e8 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:11.158625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0499f0d4-039f-4627-b24a-ccc3c3fefa6e None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:11.258185 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0499f0d4-039f-4627-b24a-ccc3c3fefa6e None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:11.291461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf2e338-316a-49e1-b8be-58022a6236c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:11.292055 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf2e338-316a-49e1-b8be-58022a6236c4 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:11.292339 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf2e338-316a-49e1-b8be-58022a6236c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:11.292606 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf2e338-316a-49e1-b8be-58022a6236c4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:11.320972 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:11.358016 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf2e338-316a-49e1-b8be-58022a6236c4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:11.359466 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf2e338-316a-49e1-b8be-58022a6236c4 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:11.452799 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 561/1128] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:49:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:49:11.459068 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6361e4a7-78c5-4bb4-8419-3000fea6ddde None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:11.556892 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6361e4a7-78c5-4bb4-8419-3000fea6ddde None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:11.590403 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64caa180-6819-4e68-a1ea-42a82bf68e13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:11.591020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64caa180-6819-4e68-a1ea-42a82bf68e13 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:11.591399 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64caa180-6819-4e68-a1ea-42a82bf68e13 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:11.591672 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64caa180-6819-4e68-a1ea-42a82bf68e13 None admin] PATH_INFO: `/v3/projects/5101d12502824233877ee3c3a7c29586` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:11.592774 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64caa180-6819-4e68-a1ea-42a82bf68e13 None admin] RBAC: Authorizing `identity:get_project(project_id=5101d12502824233877ee3c3a7c29586)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:11.594214 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64caa180-6819-4e68-a1ea-42a82bf68e13 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:11.595832 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 568/1129] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:49:11 2022] GET /identity/v3/projects/5101d12502824233877ee3c3a7c29586 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:11.602088 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf2e338-316a-49e1-b8be-58022a6236c4 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:11.661080 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:11.698363 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf2e338-316a-49e1-b8be-58022a6236c4 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:11.732025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2e4ce6-739b-4d79-9d23-d96ca5022d7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:11.732557 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2e4ce6-739b-4d79-9d23-d96ca5022d7c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:11.732835 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2e4ce6-739b-4d79-9d23-d96ca5022d7c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:11.733108 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2e4ce6-739b-4d79-9d23-d96ca5022d7c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:11.733826 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2e4ce6-739b-4d79-9d23-d96ca5022d7c None admin] RBAC: Adding query filter params (role.id=285107fb9b2449fe9f1d29f3075a246f, scope.project.id=5101d12502824233877ee3c3a7c29586, user.id=e9b5543320c942479f72653554e862e8) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:49:11.734148 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2e4ce6-739b-4d79-9d23-d96ca5022d7c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:11.735555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2e4ce6-739b-4d79-9d23-d96ca5022d7c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:11.741091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 562/1130] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:49:11 2022] GET /identity/v3/role_assignments?user.id=e9b5543320c942479f72653554e862e8&scope.project.id=5101d12502824233877ee3c3a7c29586&role.id=285107fb9b2449fe9f1d29f3075a246f => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:13.936115 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c68f114-9f1a-4d48-816c-5f1d9d836316 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:13.936645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c68f114-9f1a-4d48-816c-5f1d9d836316 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:13.936645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c68f114-9f1a-4d48-816c-5f1d9d836316 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:13.937993 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 569/1131] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:49:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:49:13.943331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2318776-fb57-46ad-b2ac-e6800aa9dc87 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:13.943720 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2318776-fb57-46ad-b2ac-e6800aa9dc87 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:13.943881 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2318776-fb57-46ad-b2ac-e6800aa9dc87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:13.962229 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-b2318776-fb57-46ad-b2ac-e6800aa9dc87 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:49:13.963434 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-b2318776-fb57-46ad-b2ac-e6800aa9dc87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:49:14.029013 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 563/1132] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:49:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:49:14.296706 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c68f114-9f1a-4d48-816c-5f1d9d836316 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:14.315896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:14.397102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c68f114-9f1a-4d48-816c-5f1d9d836316 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:14.429212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ea925c-e213-48db-9372-954911d1dd4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:14.429833 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ea925c-e213-48db-9372-954911d1dd4f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:14.430134 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ea925c-e213-48db-9372-954911d1dd4f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:14.430410 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ea925c-e213-48db-9372-954911d1dd4f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:14.431511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ea925c-e213-48db-9372-954911d1dd4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:14.433504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ea925c-e213-48db-9372-954911d1dd4f None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:14.435251 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 570/1133] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:49:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:14.441030 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2318776-fb57-46ad-b2ac-e6800aa9dc87 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:14.497203 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:14.539214 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2318776-fb57-46ad-b2ac-e6800aa9dc87 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:14.573176 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50694fc5-4a19-44a5-8e27-77cc2ff95376 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:14.573781 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50694fc5-4a19-44a5-8e27-77cc2ff95376 None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:14.574057 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50694fc5-4a19-44a5-8e27-77cc2ff95376 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:14.574355 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50694fc5-4a19-44a5-8e27-77cc2ff95376 None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:14.575007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50694fc5-4a19-44a5-8e27-77cc2ff95376 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:14.576421 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50694fc5-4a19-44a5-8e27-77cc2ff95376 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:14.601955 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 564/1134] 10.222.0.27 () {60 vars in 1141 bytes} [Wed Sep 7 03:49:14 2022] POST /identity/v3/projects => generated 311 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:49:16.746579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc85ed9-7bbf-4bd4-b070-6a6ed325b47d None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:16.747134 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc85ed9-7bbf-4bd4-b070-6a6ed325b47d None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:16.747134 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc85ed9-7bbf-4bd4-b070-6a6ed325b47d None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:16.748456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 571/1135] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:49:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:49:16.753607 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba5576a-fd72-4874-90ce-1a1b1aea99b0 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:16.754016 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba5576a-fd72-4874-90ce-1a1b1aea99b0 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:16.754193 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba5576a-fd72-4874-90ce-1a1b1aea99b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:16.772782 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-6ba5576a-fd72-4874-90ce-1a1b1aea99b0 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:49:16.773980 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-6ba5576a-fd72-4874-90ce-1a1b1aea99b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:49:16.838783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 565/1136] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:49:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:49:17.107360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc85ed9-7bbf-4bd4-b070-6a6ed325b47d None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:17.190434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:17.206601 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc85ed9-7bbf-4bd4-b070-6a6ed325b47d None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:17.239832 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f212fb-da77-4270-95a5-f252a1ba906e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:17.240456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f212fb-da77-4270-95a5-f252a1ba906e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:17.240736 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f212fb-da77-4270-95a5-f252a1ba906e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:17.241007 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f212fb-da77-4270-95a5-f252a1ba906e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:17.242109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f212fb-da77-4270-95a5-f252a1ba906e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:17.243557 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f212fb-da77-4270-95a5-f252a1ba906e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:17.245127 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 572/1137] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:49:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:17.251328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba5576a-fd72-4874-90ce-1a1b1aea99b0 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:17.289805 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:17.349967 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba5576a-fd72-4874-90ce-1a1b1aea99b0 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:17.383305 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2844de-99a4-4dcb-a2dc-ea7baba1ddf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:17.383852 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2844de-99a4-4dcb-a2dc-ea7baba1ddf3 None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:17.384136 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2844de-99a4-4dcb-a2dc-ea7baba1ddf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:17.384424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2844de-99a4-4dcb-a2dc-ea7baba1ddf3 None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:17.385056 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2844de-99a4-4dcb-a2dc-ea7baba1ddf3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:17.387085 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2844de-99a4-4dcb-a2dc-ea7baba1ddf3 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:17.407176 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 566/1138] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:49:17 2022] POST /identity/v3/users => generated 286 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:49:19.572781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b31bea7-a51e-4af0-8657-cbccc3f29eee None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:19.573392 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b31bea7-a51e-4af0-8657-cbccc3f29eee None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:19.573392 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b31bea7-a51e-4af0-8657-cbccc3f29eee None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:19.574670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 573/1139] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:49:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:49:19.580090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85a022a-1324-4b74-868b-90f5439ade47 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:19.580461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85a022a-1324-4b74-868b-90f5439ade47 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:19.580672 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85a022a-1324-4b74-868b-90f5439ade47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:19.598813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-f85a022a-1324-4b74-868b-90f5439ade47 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:49:19.599984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-f85a022a-1324-4b74-868b-90f5439ade47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:49:19.664352 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 567/1140] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:49:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:49:19.930315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b31bea7-a51e-4af0-8657-cbccc3f29eee None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:20.029609 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b31bea7-a51e-4af0-8657-cbccc3f29eee None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:20.061577 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd89896-d78b-464b-a555-f0a3e8dc0a75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:20.062222 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd89896-d78b-464b-a555-f0a3e8dc0a75 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:20.062501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd89896-d78b-464b-a555-f0a3e8dc0a75 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:20.062772 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd89896-d78b-464b-a555-f0a3e8dc0a75 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:20.072449 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd89896-d78b-464b-a555-f0a3e8dc0a75 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:49:20.077781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-2dd89896-d78b-464b-a555-f0a3e8dc0a75 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:49:20.077781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:49:20.077781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:49:20.077781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:49:20.077781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:49:20.077781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:49:20.077781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:49:20.077781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:49:20.077781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:49:20.077781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:49:20.077781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:49:20.077781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:49:20.077781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:49:20.077781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:49:20.077781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:49:20.077781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:49:20.078750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:49:20.078750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:49:20.078750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:49:20.078750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:49:20.078750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:49:20.078750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:49:20.078750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:49:20.078750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:49:20.078750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:49:20.078750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:49:20.078750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:49:20.078750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:49:20.078750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:49:20.078750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:49:20.078750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:49:20.078750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:49:20.079992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:49:20.079992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:49:20.079992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:49:20.079992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:49:20.079992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:49:20.079992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:49:20.079992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:49:20.079992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:49:20.079992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:49:20.079992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:49:20.079992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:49:20.079992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:49:20.079992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 574/1141] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:49:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:49:20.085090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85a022a-1324-4b74-868b-90f5439ade47 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:20.109173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:20.183343 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85a022a-1324-4b74-868b-90f5439ade47 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:20.216569 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1dcee9-2bbc-4ae4-839a-30dab05b6cad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:20.217169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1dcee9-2bbc-4ae4-839a-30dab05b6cad None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:20.217451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1dcee9-2bbc-4ae4-839a-30dab05b6cad None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:20.217726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1dcee9-2bbc-4ae4-839a-30dab05b6cad None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:20.218420 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1dcee9-2bbc-4ae4-839a-30dab05b6cad None admin] RBAC: Adding query filter params (name=admin) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:49:20.218727 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1dcee9-2bbc-4ae4-839a-30dab05b6cad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:20.220086 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1dcee9-2bbc-4ae4-839a-30dab05b6cad None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:20.227863 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 568/1142] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:49:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:20.233954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd89896-d78b-464b-a555-f0a3e8dc0a75 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:20.250395 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:20.333377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd89896-d78b-464b-a555-f0a3e8dc0a75 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:20.365627 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56492638-cf82-4f04-a989-2983af8214c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:20.366282 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56492638-cf82-4f04-a989-2983af8214c0 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:20.366566 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56492638-cf82-4f04-a989-2983af8214c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:20.366841 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56492638-cf82-4f04-a989-2983af8214c0 None admin] PATH_INFO: `/v3/users/c3ee6246c27b4126b67c43370c79d545` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:20.380328 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56492638-cf82-4f04-a989-2983af8214c0 None admin] RBAC: Authorizing `identity:get_user(user_id=c3ee6246c27b4126b67c43370c79d545)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:20.381768 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56492638-cf82-4f04-a989-2983af8214c0 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:20.383397 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 575/1143] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:49:20 2022] GET /identity/v3/users/c3ee6246c27b4126b67c43370c79d545 => generated 286 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:20.389972 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1dcee9-2bbc-4ae4-839a-30dab05b6cad None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:20.472879 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:20.488548 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1dcee9-2bbc-4ae4-839a-30dab05b6cad None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:20.522105 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05405776-cc31-4dab-afd4-310ba86dae34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:20.522730 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05405776-cc31-4dab-afd4-310ba86dae34 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:20.523008 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05405776-cc31-4dab-afd4-310ba86dae34 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:20.523279 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05405776-cc31-4dab-afd4-310ba86dae34 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:20.588525 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05405776-cc31-4dab-afd4-310ba86dae34 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:20.590057 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05405776-cc31-4dab-afd4-310ba86dae34 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:20.670010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:20.686132 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 569/1144] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:49:20 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:49:20.692211 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56492638-cf82-4f04-a989-2983af8214c0 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:20.761231 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:20.789588 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56492638-cf82-4f04-a989-2983af8214c0 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:20.823276 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6598fed-de06-41e6-8b02-36e329719873 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:20.824253 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6598fed-de06-41e6-8b02-36e329719873 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:20.824541 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6598fed-de06-41e6-8b02-36e329719873 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:20.824820 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6598fed-de06-41e6-8b02-36e329719873 None admin] PATH_INFO: `/v3/projects/393c2b47bdae4e369a22e37061074266` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:20.825937 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6598fed-de06-41e6-8b02-36e329719873 None admin] RBAC: Authorizing `identity:get_project(project_id=393c2b47bdae4e369a22e37061074266)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:20.827423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6598fed-de06-41e6-8b02-36e329719873 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:20.829064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 576/1145] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:49:20 2022] GET /identity/v3/projects/393c2b47bdae4e369a22e37061074266 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:20.835572 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05405776-cc31-4dab-afd4-310ba86dae34 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:20.931717 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05405776-cc31-4dab-afd4-310ba86dae34 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:20.964978 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e6c3a7-b325-492b-b921-69cd89e3d557 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:20.965561 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e6c3a7-b325-492b-b921-69cd89e3d557 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:20.965833 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e6c3a7-b325-492b-b921-69cd89e3d557 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:20.966099 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e6c3a7-b325-492b-b921-69cd89e3d557 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:20.966833 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e6c3a7-b325-492b-b921-69cd89e3d557 None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=393c2b47bdae4e369a22e37061074266, user.id=c3ee6246c27b4126b67c43370c79d545) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:49:20.967133 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e6c3a7-b325-492b-b921-69cd89e3d557 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:20.968530 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e6c3a7-b325-492b-b921-69cd89e3d557 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:20.973655 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 570/1146] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:49:20 2022] GET /identity/v3/role_assignments?user.id=c3ee6246c27b4126b67c43370c79d545&scope.project.id=393c2b47bdae4e369a22e37061074266&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:23.102012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e6587b-ac92-4489-8189-d67f72301afb None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:23.102574 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e6587b-ac92-4489-8189-d67f72301afb None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:23.102574 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e6587b-ac92-4489-8189-d67f72301afb None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:23.103920 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 577/1147] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:49:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:49:23.109309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28eaf023-842c-48d7-8cce-9ff3bf00f866 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:23.109713 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28eaf023-842c-48d7-8cce-9ff3bf00f866 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:23.109856 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28eaf023-842c-48d7-8cce-9ff3bf00f866 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:23.128260 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-28eaf023-842c-48d7-8cce-9ff3bf00f866 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:49:23.129440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-28eaf023-842c-48d7-8cce-9ff3bf00f866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:49:23.173832 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:23.194563 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 571/1148] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:49:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:49:23.463712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e6587b-ac92-4489-8189-d67f72301afb None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:23.560969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e6587b-ac92-4489-8189-d67f72301afb None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:23.595066 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400e54da-18bd-497b-bcf5-ac9f78b50172 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:23.595679 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400e54da-18bd-497b-bcf5-ac9f78b50172 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:23.595952 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400e54da-18bd-497b-bcf5-ac9f78b50172 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:23.596217 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400e54da-18bd-497b-bcf5-ac9f78b50172 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:23.603394 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:23.606722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400e54da-18bd-497b-bcf5-ac9f78b50172 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:49:23.611639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-400e54da-18bd-497b-bcf5-ac9f78b50172 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:49:23.611639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:49:23.611639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:49:23.611639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:49:23.611639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:49:23.611639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:49:23.611639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:49:23.611639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:49:23.611639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:49:23.611639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:49:23.611639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:49:23.611639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:49:23.611639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:49:23.611639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:49:23.611639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:49:23.611639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:49:23.612731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:49:23.612731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:49:23.612731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:49:23.612731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:49:23.612731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:49:23.612731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:49:23.612731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:49:23.612731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:49:23.612731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:49:23.612731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:49:23.612731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:49:23.612731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:49:23.612731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:49:23.612731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:49:23.612731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:49:23.612731 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:49:23.613946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:49:23.613946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:49:23.613946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:49:23.613946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:49:23.613946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:49:23.613946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:49:23.613946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:49:23.613946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:49:23.613946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:49:23.613946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:49:23.613946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:49:23.613946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:49:23.613946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 578/1149] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:49:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:49:23.618667 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28eaf023-842c-48d7-8cce-9ff3bf00f866 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:23.713190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28eaf023-842c-48d7-8cce-9ff3bf00f866 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:23.746190 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932686a2-41f0-4ed8-b7e1-24ed327d8ca8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:23.746742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932686a2-41f0-4ed8-b7e1-24ed327d8ca8 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:23.747020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932686a2-41f0-4ed8-b7e1-24ed327d8ca8 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:23.747289 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932686a2-41f0-4ed8-b7e1-24ed327d8ca8 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:23.747944 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932686a2-41f0-4ed8-b7e1-24ed327d8ca8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:49:23.748244 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932686a2-41f0-4ed8-b7e1-24ed327d8ca8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:23.749594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932686a2-41f0-4ed8-b7e1-24ed327d8ca8 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:23.757305 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 572/1150] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:49:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:23.763819 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400e54da-18bd-497b-bcf5-ac9f78b50172 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:23.860797 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400e54da-18bd-497b-bcf5-ac9f78b50172 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:23.893814 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd80b137-e293-417a-92c2-cd5b5259c074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:23.894458 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd80b137-e293-417a-92c2-cd5b5259c074 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:23.894738 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd80b137-e293-417a-92c2-cd5b5259c074 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:23.895013 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd80b137-e293-417a-92c2-cd5b5259c074 None admin] PATH_INFO: `/v3/users/c3ee6246c27b4126b67c43370c79d545` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:23.896114 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd80b137-e293-417a-92c2-cd5b5259c074 None admin] RBAC: Authorizing `identity:get_user(user_id=c3ee6246c27b4126b67c43370c79d545)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:23.897535 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd80b137-e293-417a-92c2-cd5b5259c074 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:23.899115 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 579/1151] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:49:23 2022] GET /identity/v3/users/c3ee6246c27b4126b67c43370c79d545 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:23.904819 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932686a2-41f0-4ed8-b7e1-24ed327d8ca8 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:23.947113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:24.003004 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932686a2-41f0-4ed8-b7e1-24ed327d8ca8 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:24.035291 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8590ac-2303-43e1-a1dd-12f70b05f2ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:24.035905 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8590ac-2303-43e1-a1dd-12f70b05f2ba None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:24.036187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8590ac-2303-43e1-a1dd-12f70b05f2ba None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:24.036464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8590ac-2303-43e1-a1dd-12f70b05f2ba None admin] PATH_INFO: `/v3/projects/393c2b47bdae4e369a22e37061074266` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:24.037579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8590ac-2303-43e1-a1dd-12f70b05f2ba None admin] RBAC: Authorizing `identity:get_project(project_id=393c2b47bdae4e369a22e37061074266)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:24.039032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8590ac-2303-43e1-a1dd-12f70b05f2ba None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:24.040636 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 573/1152] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:49:23 2022] GET /identity/v3/projects/393c2b47bdae4e369a22e37061074266 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:24.046841 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd80b137-e293-417a-92c2-cd5b5259c074 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:24.112286 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:24.144235 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd80b137-e293-417a-92c2-cd5b5259c074 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:24.176706 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf56736-523b-4bd2-8a5c-76218bdac280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:24.177350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf56736-523b-4bd2-8a5c-76218bdac280 None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:24.177627 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf56736-523b-4bd2-8a5c-76218bdac280 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:24.177900 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf56736-523b-4bd2-8a5c-76218bdac280 None admin] PATH_INFO: `/v3/projects/393c2b47bdae4e369a22e37061074266/users/c3ee6246c27b4126b67c43370c79d545/roles/9d5b19a61c46475d8048b1bd66ca25a1` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:24.179798 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf56736-523b-4bd2-8a5c-76218bdac280 None admin] RBAC: Authorizing `identity:create_grant(project_id=393c2b47bdae4e369a22e37061074266, user_id=c3ee6246c27b4126b67c43370c79d545, role_id=9d5b19a61c46475d8048b1bd66ca25a1)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:24.181222 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf56736-523b-4bd2-8a5c-76218bdac280 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:24.194363 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 580/1153] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:49:24 2022] PUT /identity/v3/projects/393c2b47bdae4e369a22e37061074266/users/c3ee6246c27b4126b67c43370c79d545/roles/9d5b19a61c46475d8048b1bd66ca25a1 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:49:26.314512 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c47bd96-b425-4174-925b-5f5de16109ae None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:26.315053 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c47bd96-b425-4174-925b-5f5de16109ae None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:26.315053 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c47bd96-b425-4174-925b-5f5de16109ae None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:26.316373 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 574/1154] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:49:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:49:26.321773 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a998c9-0e5b-436e-91db-2632f4d43a7b None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:26.322186 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a998c9-0e5b-436e-91db-2632f4d43a7b None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:26.322357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a998c9-0e5b-436e-91db-2632f4d43a7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:26.340998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-23a998c9-0e5b-436e-91db-2632f4d43a7b None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:49:26.342203 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-23a998c9-0e5b-436e-91db-2632f4d43a7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:49:26.406615 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 581/1155] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:49:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:49:26.677733 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c47bd96-b425-4174-925b-5f5de16109ae None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:26.777423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c47bd96-b425-4174-925b-5f5de16109ae None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:26.787587 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:26.811528 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a09fc8-a8a6-4853-8d43-501033f01d79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:26.812170 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a09fc8-a8a6-4853-8d43-501033f01d79 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:26.812467 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a09fc8-a8a6-4853-8d43-501033f01d79 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:26.812723 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a09fc8-a8a6-4853-8d43-501033f01d79 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:26.822447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a09fc8-a8a6-4853-8d43-501033f01d79 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:49:26.827506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-68a09fc8-a8a6-4853-8d43-501033f01d79 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:49:26.827506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:49:26.827506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:49:26.827506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:49:26.827506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:49:26.827506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:49:26.827506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:49:26.827506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:49:26.827506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:49:26.827506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:49:26.827506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:49:26.827506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:49:26.827506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:49:26.827506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:49:26.827506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:49:26.827506 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:49:26.828679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:49:26.828679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:49:26.828679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:49:26.828679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:49:26.828679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:49:26.828679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:49:26.828679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:49:26.828679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:49:26.828679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:49:26.828679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:49:26.828679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:49:26.828679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:49:26.828679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:49:26.828679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:49:26.828679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:49:26.828679 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:49:26.830247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:49:26.830247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:49:26.830247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:49:26.830247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:49:26.830247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:49:26.830247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:49:26.830247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:49:26.830247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:49:26.830247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:49:26.830247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:49:26.830247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:49:26.830247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:49:26.830247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 575/1156] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:49:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:49:26.834933 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a998c9-0e5b-436e-91db-2632f4d43a7b None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:26.891239 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:26.933682 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a998c9-0e5b-436e-91db-2632f4d43a7b None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:26.972620 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8b16fc-3c9e-47b6-a61a-e357fb81271b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:26.973221 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8b16fc-3c9e-47b6-a61a-e357fb81271b None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:26.973501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8b16fc-3c9e-47b6-a61a-e357fb81271b None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:26.973778 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8b16fc-3c9e-47b6-a61a-e357fb81271b None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:26.974466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8b16fc-3c9e-47b6-a61a-e357fb81271b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:49:26.974770 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8b16fc-3c9e-47b6-a61a-e357fb81271b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:26.976130 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8b16fc-3c9e-47b6-a61a-e357fb81271b None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:26.984428 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 582/1157] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:49:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:26.990406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a09fc8-a8a6-4853-8d43-501033f01d79 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:27.086785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a09fc8-a8a6-4853-8d43-501033f01d79 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:27.119912 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117e7404-404a-4a49-9bff-7d1a645c9a94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:27.120608 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117e7404-404a-4a49-9bff-7d1a645c9a94 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:27.120883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117e7404-404a-4a49-9bff-7d1a645c9a94 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:27.121153 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117e7404-404a-4a49-9bff-7d1a645c9a94 None admin] PATH_INFO: `/v3/users/c3ee6246c27b4126b67c43370c79d545` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:27.122292 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117e7404-404a-4a49-9bff-7d1a645c9a94 None admin] RBAC: Authorizing `identity:get_user(user_id=c3ee6246c27b4126b67c43370c79d545)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:27.123705 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117e7404-404a-4a49-9bff-7d1a645c9a94 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:27.125246 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 576/1158] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:49:26 2022] GET /identity/v3/users/c3ee6246c27b4126b67c43370c79d545 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:27.130913 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8b16fc-3c9e-47b6-a61a-e357fb81271b None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:27.227852 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8b16fc-3c9e-47b6-a61a-e357fb81271b None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:27.247256 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:27.261867 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfac6480-b19f-484d-a793-918bd0f9deb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:27.262507 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfac6480-b19f-484d-a793-918bd0f9deb6 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:27.262789 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfac6480-b19f-484d-a793-918bd0f9deb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:27.263062 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfac6480-b19f-484d-a793-918bd0f9deb6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:27.328487 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfac6480-b19f-484d-a793-918bd0f9deb6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:27.330215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfac6480-b19f-484d-a793-918bd0f9deb6 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:27.424529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 583/1159] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:49:27 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:49:27.430491 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117e7404-404a-4a49-9bff-7d1a645c9a94 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:27.470514 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:27.528783 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117e7404-404a-4a49-9bff-7d1a645c9a94 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:27.561053 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf6406e-a351-4510-b1bc-f972a953aaa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:27.561675 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf6406e-a351-4510-b1bc-f972a953aaa5 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:27.561959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf6406e-a351-4510-b1bc-f972a953aaa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:27.562260 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf6406e-a351-4510-b1bc-f972a953aaa5 None admin] PATH_INFO: `/v3/projects/393c2b47bdae4e369a22e37061074266` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:27.563379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf6406e-a351-4510-b1bc-f972a953aaa5 None admin] RBAC: Authorizing `identity:get_project(project_id=393c2b47bdae4e369a22e37061074266)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:27.564790 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf6406e-a351-4510-b1bc-f972a953aaa5 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:27.566438 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 577/1160] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:49:27 2022] GET /identity/v3/projects/393c2b47bdae4e369a22e37061074266 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:27.572547 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfac6480-b19f-484d-a793-918bd0f9deb6 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:27.596734 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:27.671927 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfac6480-b19f-484d-a793-918bd0f9deb6 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:27.705436 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0e4459-a761-407e-8bc0-6f950bfdb0f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:27.706027 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0e4459-a761-407e-8bc0-6f950bfdb0f7 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:27.706331 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0e4459-a761-407e-8bc0-6f950bfdb0f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:27.706600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0e4459-a761-407e-8bc0-6f950bfdb0f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:27.707308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0e4459-a761-407e-8bc0-6f950bfdb0f7 None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=393c2b47bdae4e369a22e37061074266, user.id=c3ee6246c27b4126b67c43370c79d545) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:49:27.707609 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0e4459-a761-407e-8bc0-6f950bfdb0f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:27.709016 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0e4459-a761-407e-8bc0-6f950bfdb0f7 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:27.714819 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 584/1161] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:49:27 2022] GET /identity/v3/role_assignments?user.id=c3ee6246c27b4126b67c43370c79d545&scope.project.id=393c2b47bdae4e369a22e37061074266&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:29.876856 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41255b0-15d8-4768-a13c-34c018554112 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:29.877413 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41255b0-15d8-4768-a13c-34c018554112 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:29.877413 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41255b0-15d8-4768-a13c-34c018554112 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:29.878760 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 578/1162] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:49:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:49:29.884005 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca3d01b-3574-4ca6-96e2-d8face15c107 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:29.884393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca3d01b-3574-4ca6-96e2-d8face15c107 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:29.884557 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca3d01b-3574-4ca6-96e2-d8face15c107 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:29.903039 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-9ca3d01b-3574-4ca6-96e2-d8face15c107 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:49:29.904229 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-9ca3d01b-3574-4ca6-96e2-d8face15c107 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:49:29.969417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 585/1163] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:49:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:49:30.235734 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41255b0-15d8-4768-a13c-34c018554112 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:30.334881 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41255b0-15d8-4768-a13c-34c018554112 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:30.342562 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:30.368482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b4fcc7-305e-4581-aae4-1e2ffb946fe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:30.369076 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b4fcc7-305e-4581-aae4-1e2ffb946fe1 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:30.369353 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b4fcc7-305e-4581-aae4-1e2ffb946fe1 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:30.369626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b4fcc7-305e-4581-aae4-1e2ffb946fe1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:30.434493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b4fcc7-305e-4581-aae4-1e2ffb946fe1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:30.435925 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b4fcc7-305e-4581-aae4-1e2ffb946fe1 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:30.531666 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 579/1164] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:49:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:49:30.537942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca3d01b-3574-4ca6-96e2-d8face15c107 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:30.553264 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:30.635923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca3d01b-3574-4ca6-96e2-d8face15c107 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:30.669619 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4629178e-9bfd-489c-bc93-b00ad2e169ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:30.670284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4629178e-9bfd-489c-bc93-b00ad2e169ce None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:30.670563 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4629178e-9bfd-489c-bc93-b00ad2e169ce None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:30.670834 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4629178e-9bfd-489c-bc93-b00ad2e169ce None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:30.677240 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4629178e-9bfd-489c-bc93-b00ad2e169ce None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:30.678707 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4629178e-9bfd-489c-bc93-b00ad2e169ce None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-4629178e-9bfd-489c-bc93-b00ad2e169ce None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:49:30.689749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:49:30.691059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:49:30.691059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:49:30.691059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:49:30.691059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:49:30.691059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:49:30.691059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:49:30.691059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:49:30.691059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:49:30.691059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:49:30.691059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:49:30.691059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:49:30.691059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:49:30.691059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:49:30.691059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:49:30.691059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:49:30.691059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:49:30.692219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:49:30.692219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:49:30.692219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:49:30.692219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:49:30.692219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:49:30.692219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:49:30.692219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:49:30.692219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:49:30.692219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:49:30.692219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:49:30.692219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 07 03:49:30.692219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:49:30.692219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 07 03:49:30.692219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:49:30.692219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 586/1165] 10.222.0.27 () {56 vars in 1117 bytes} [Wed Sep 7 03:49:30 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:49:30.696909 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b4fcc7-305e-4581-aae4-1e2ffb946fe1 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:30.705580 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:30.795364 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b4fcc7-305e-4581-aae4-1e2ffb946fe1 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:30.827517 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dffa7d9-780c-4d46-8e48-31ea106797d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:30.828111 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dffa7d9-780c-4d46-8e48-31ea106797d6 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:30.828393 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dffa7d9-780c-4d46-8e48-31ea106797d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:30.828667 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dffa7d9-780c-4d46-8e48-31ea106797d6 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:30.829558 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dffa7d9-780c-4d46-8e48-31ea106797d6 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:49:30.829870 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dffa7d9-780c-4d46-8e48-31ea106797d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:30.831250 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dffa7d9-780c-4d46-8e48-31ea106797d6 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:30.838359 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 580/1166] 10.222.0.27 () {56 vars in 1115 bytes} [Wed Sep 7 03:49:30 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:32.965809 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46edc62-98ba-4624-b727-9dea7270d069 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:32.966398 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46edc62-98ba-4624-b727-9dea7270d069 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:32.966398 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46edc62-98ba-4624-b727-9dea7270d069 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:32.967732 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 587/1167] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:49:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:49:32.972887 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc77fd1c-65d6-4086-a0b6-46412a4cec84 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:32.973303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc77fd1c-65d6-4086-a0b6-46412a4cec84 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:32.973430 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc77fd1c-65d6-4086-a0b6-46412a4cec84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:32.991799 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-cc77fd1c-65d6-4086-a0b6-46412a4cec84 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:49:32.992985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-cc77fd1c-65d6-4086-a0b6-46412a4cec84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:49:33.054080 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:33.059180 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 581/1168] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:49:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:49:33.332987 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46edc62-98ba-4624-b727-9dea7270d069 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:33.407696 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:33.432690 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46edc62-98ba-4624-b727-9dea7270d069 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:33.465441 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32edc48b-10e5-4d0d-96f5-efb0ac0f4040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:33.466031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32edc48b-10e5-4d0d-96f5-efb0ac0f4040 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:33.466337 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32edc48b-10e5-4d0d-96f5-efb0ac0f4040 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:33.466606 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32edc48b-10e5-4d0d-96f5-efb0ac0f4040 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:33.467271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32edc48b-10e5-4d0d-96f5-efb0ac0f4040 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:33.468623 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32edc48b-10e5-4d0d-96f5-efb0ac0f4040 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:33.483948 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 588/1169] 10.222.0.27 () {60 vars in 1138 bytes} [Wed Sep 7 03:49:33 2022] POST /identity/v3/domains => generated 267 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:49:35.780801 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6468ced-a57d-4bd4-a0d0-9572d2dcaa8d None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:35.781304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6468ced-a57d-4bd4-a0d0-9572d2dcaa8d None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:35.781304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6468ced-a57d-4bd4-a0d0-9572d2dcaa8d None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:35.782701 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 582/1170] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:49:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:49:35.787857 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc4df01-4981-4c61-9edb-a93d3fec887e None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:35.788258 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc4df01-4981-4c61-9edb-a93d3fec887e None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:35.788405 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc4df01-4981-4c61-9edb-a93d3fec887e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:35.808717 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-bfc4df01-4981-4c61-9edb-a93d3fec887e None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:49:35.809900 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-bfc4df01-4981-4c61-9edb-a93d3fec887e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:49:35.875277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 589/1171] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:49:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:49:36.144303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6468ced-a57d-4bd4-a0d0-9572d2dcaa8d None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:36.241838 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6468ced-a57d-4bd4-a0d0-9572d2dcaa8d None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:36.275053 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be45bd9-2783-454f-92a4-d3b14b80aeaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:36.275670 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be45bd9-2783-454f-92a4-d3b14b80aeaf None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:36.276000 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be45bd9-2783-454f-92a4-d3b14b80aeaf None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:36.276786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be45bd9-2783-454f-92a4-d3b14b80aeaf None admin] PATH_INFO: `/v3/domains/5283a247b57d4a7cbe8fd03c589f0ce6` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:36.286790 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be45bd9-2783-454f-92a4-d3b14b80aeaf None admin] RBAC: Authorizing `identity:get_domain(domain_id=5283a247b57d4a7cbe8fd03c589f0ce6)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:36.288206 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be45bd9-2783-454f-92a4-d3b14b80aeaf None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:36.289760 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 583/1172] 10.222.0.27 () {56 vars in 1183 bytes} [Wed Sep 7 03:49:36 2022] GET /identity/v3/domains/5283a247b57d4a7cbe8fd03c589f0ce6 => generated 267 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:36.295793 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc4df01-4981-4c61-9edb-a93d3fec887e None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:36.355552 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:36.392359 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc4df01-4981-4c61-9edb-a93d3fec887e None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:36.425001 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a013c29-3164-4f85-956d-badb2b34af41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:36.425544 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a013c29-3164-4f85-956d-badb2b34af41 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:36.425823 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a013c29-3164-4f85-956d-badb2b34af41 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:36.426095 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a013c29-3164-4f85-956d-badb2b34af41 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:36.426761 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a013c29-3164-4f85-956d-badb2b34af41 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:36.428199 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a013c29-3164-4f85-956d-badb2b34af41 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:36.453646 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 590/1173] 10.222.0.27 () {60 vars in 1141 bytes} [Wed Sep 7 03:49:36 2022] POST /identity/v3/projects => generated 361 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:49:38.628992 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291db595-8a01-4692-9ae9-f562d8742f24 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:38.629521 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291db595-8a01-4692-9ae9-f562d8742f24 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:38.629521 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291db595-8a01-4692-9ae9-f562d8742f24 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:38.630915 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 584/1174] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:49:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:49:38.637065 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a106f2d7-8836-4974-844a-cf6b3944ec4a None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:38.637475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a106f2d7-8836-4974-844a-cf6b3944ec4a None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:38.637623 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a106f2d7-8836-4974-844a-cf6b3944ec4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:38.656344 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-a106f2d7-8836-4974-844a-cf6b3944ec4a None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:49:38.657536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-a106f2d7-8836-4974-844a-cf6b3944ec4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:49:38.723071 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 591/1175] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:49:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:49:38.994473 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291db595-8a01-4692-9ae9-f562d8742f24 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:39.053503 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:39.096066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291db595-8a01-4692-9ae9-f562d8742f24 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:39.128950 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276456b3-a59e-4e3c-9058-68dd310525f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:39.129565 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276456b3-a59e-4e3c-9058-68dd310525f2 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:39.129843 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276456b3-a59e-4e3c-9058-68dd310525f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:39.130149 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276456b3-a59e-4e3c-9058-68dd310525f2 None admin] PATH_INFO: `/v3/domains/5283a247b57d4a7cbe8fd03c589f0ce6` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:39.131254 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276456b3-a59e-4e3c-9058-68dd310525f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=5283a247b57d4a7cbe8fd03c589f0ce6)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:39.132671 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276456b3-a59e-4e3c-9058-68dd310525f2 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:39.134236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 585/1176] 10.222.0.27 () {56 vars in 1183 bytes} [Wed Sep 7 03:49:38 2022] GET /identity/v3/domains/5283a247b57d4a7cbe8fd03c589f0ce6 => generated 267 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:39.140533 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a106f2d7-8836-4974-844a-cf6b3944ec4a None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:39.185017 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:39.236845 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a106f2d7-8836-4974-844a-cf6b3944ec4a None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:39.270096 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05edadd1-48e3-450c-b13b-8b0a997d94f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:39.270725 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05edadd1-48e3-450c-b13b-8b0a997d94f2 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:39.271003 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05edadd1-48e3-450c-b13b-8b0a997d94f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:39.271273 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05edadd1-48e3-450c-b13b-8b0a997d94f2 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:39.271909 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05edadd1-48e3-450c-b13b-8b0a997d94f2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:39.273335 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05edadd1-48e3-450c-b13b-8b0a997d94f2 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:39.294193 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 592/1177] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:49:39 2022] POST /identity/v3/users => generated 311 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:49:41.480147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690577cd-61b1-4906-8a46-cdf633743289 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:41.480695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690577cd-61b1-4906-8a46-cdf633743289 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:41.480695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690577cd-61b1-4906-8a46-cdf633743289 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:41.482933 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 586/1178] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:49:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:49:41.488264 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532867f2-81f3-4ac3-a9b1-47b5530a9e27 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:41.488699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532867f2-81f3-4ac3-a9b1-47b5530a9e27 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:41.488819 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532867f2-81f3-4ac3-a9b1-47b5530a9e27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:41.506836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-532867f2-81f3-4ac3-a9b1-47b5530a9e27 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:49:41.508020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-532867f2-81f3-4ac3-a9b1-47b5530a9e27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:49:41.573230 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 593/1179] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:49:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:49:41.842078 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690577cd-61b1-4906-8a46-cdf633743289 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:41.939394 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690577cd-61b1-4906-8a46-cdf633743289 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:41.965179 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:41.973395 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015cb1f0-3104-43f4-a158-3173fdffffc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:41.974018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015cb1f0-3104-43f4-a158-3173fdffffc5 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:41.974324 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015cb1f0-3104-43f4-a158-3173fdffffc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:41.974598 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015cb1f0-3104-43f4-a158-3173fdffffc5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:41.985020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015cb1f0-3104-43f4-a158-3173fdffffc5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:49:41.989874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-015cb1f0-3104-43f4-a158-3173fdffffc5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:49:41.989874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:49:41.989874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:49:41.989874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:49:41.989874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:49:41.989874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:49:41.989874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:49:41.989874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:49:41.989874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:49:41.989874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:49:41.989874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:49:41.989874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:49:41.989874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:49:41.989874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:49:41.989874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:49:41.989874 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:49:41.991009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:49:41.991009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:49:41.991009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:49:41.991009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:49:41.991009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:49:41.991009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:49:41.991009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:49:41.991009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:49:41.991009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:49:41.991009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:49:41.991009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:49:41.991009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:49:41.991009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:49:41.991009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:49:41.991009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:49:41.991009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:49:41.991883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:49:41.991883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:49:41.991883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:49:41.991883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:49:41.991883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:49:41.991883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:49:41.991883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:49:41.991883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:49:41.991883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:49:41.991883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:49:41.991883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:49:41.991883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:49:41.991883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 587/1180] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:49:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:49:41.997020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532867f2-81f3-4ac3-a9b1-47b5530a9e27 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:42.026849 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:42.093992 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532867f2-81f3-4ac3-a9b1-47b5530a9e27 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:42.127351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa000d4-e64d-4258-9960-be55092571f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:42.127952 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa000d4-e64d-4258-9960-be55092571f7 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:42.128233 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa000d4-e64d-4258-9960-be55092571f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:42.128508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa000d4-e64d-4258-9960-be55092571f7 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:42.129172 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa000d4-e64d-4258-9960-be55092571f7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:49:42.129477 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa000d4-e64d-4258-9960-be55092571f7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:42.130872 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa000d4-e64d-4258-9960-be55092571f7 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:42.138889 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 594/1181] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:49:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:42.144980 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015cb1f0-3104-43f4-a158-3173fdffffc5 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:42.244387 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015cb1f0-3104-43f4-a158-3173fdffffc5 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:42.278401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3e7787-9b63-42c1-8d8b-eccfd1a0517f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:42.279023 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3e7787-9b63-42c1-8d8b-eccfd1a0517f None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:42.279305 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3e7787-9b63-42c1-8d8b-eccfd1a0517f None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:42.279580 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3e7787-9b63-42c1-8d8b-eccfd1a0517f None admin] PATH_INFO: `/v3/users/083360f37e104e8e8ebd4fa02ea214ad` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:42.292417 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3e7787-9b63-42c1-8d8b-eccfd1a0517f None admin] RBAC: Authorizing `identity:get_user(user_id=083360f37e104e8e8ebd4fa02ea214ad)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:42.293842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3e7787-9b63-42c1-8d8b-eccfd1a0517f None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:42.295464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 588/1182] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:49:42 2022] GET /identity/v3/users/083360f37e104e8e8ebd4fa02ea214ad => generated 311 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:42.301217 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa000d4-e64d-4258-9960-be55092571f7 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:42.393101 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:42.399898 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa000d4-e64d-4258-9960-be55092571f7 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:42.432610 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0703ec8c-10f7-4f52-8403-cb77f04d6d81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:42.433286 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0703ec8c-10f7-4f52-8403-cb77f04d6d81 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:42.433566 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0703ec8c-10f7-4f52-8403-cb77f04d6d81 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:42.433840 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0703ec8c-10f7-4f52-8403-cb77f04d6d81 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:42.498696 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0703ec8c-10f7-4f52-8403-cb77f04d6d81 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:42.500126 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0703ec8c-10f7-4f52-8403-cb77f04d6d81 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:42.588892 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:42.596331 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 595/1183] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:49:42 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:49:42.602297 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3e7787-9b63-42c1-8d8b-eccfd1a0517f None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:42.647952 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:42.700173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3e7787-9b63-42c1-8d8b-eccfd1a0517f None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:42.734789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17ed266-87df-41c4-9bb4-a293ead9115d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:42.735408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17ed266-87df-41c4-9bb4-a293ead9115d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:42.735687 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17ed266-87df-41c4-9bb4-a293ead9115d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:42.735961 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17ed266-87df-41c4-9bb4-a293ead9115d None admin] PATH_INFO: `/v3/projects/882877ae7aca430d91b9e4d0f2f015c0` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:42.737076 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17ed266-87df-41c4-9bb4-a293ead9115d None admin] RBAC: Authorizing `identity:get_project(project_id=882877ae7aca430d91b9e4d0f2f015c0)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:42.738536 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17ed266-87df-41c4-9bb4-a293ead9115d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:42.740152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 589/1184] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:49:42 2022] GET /identity/v3/projects/882877ae7aca430d91b9e4d0f2f015c0 => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:42.746641 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0703ec8c-10f7-4f52-8403-cb77f04d6d81 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:42.841538 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0703ec8c-10f7-4f52-8403-cb77f04d6d81 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:42.874092 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f599cd78-f31c-463c-b78c-e488ee2b407b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:42.874719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f599cd78-f31c-463c-b78c-e488ee2b407b None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:42.874994 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f599cd78-f31c-463c-b78c-e488ee2b407b None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:42.875267 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f599cd78-f31c-463c-b78c-e488ee2b407b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:42.875980 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f599cd78-f31c-463c-b78c-e488ee2b407b None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=882877ae7aca430d91b9e4d0f2f015c0, user.id=083360f37e104e8e8ebd4fa02ea214ad) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:49:42.876290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f599cd78-f31c-463c-b78c-e488ee2b407b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:42.877711 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f599cd78-f31c-463c-b78c-e488ee2b407b None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:42.882971 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 596/1185] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:49:42 2022] GET /identity/v3/role_assignments?user.id=083360f37e104e8e8ebd4fa02ea214ad&scope.project.id=882877ae7aca430d91b9e4d0f2f015c0&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:44.981175 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f528b0-0cef-4a97-8cfe-f2e33ee0ab4e None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:44.981708 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f528b0-0cef-4a97-8cfe-f2e33ee0ab4e None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:44.981823 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f528b0-0cef-4a97-8cfe-f2e33ee0ab4e None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:44.983113 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 590/1186] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:49:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:49:44.988440 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad012c0-a9b6-4ca9-9ebe-b2d505633b2c None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:44.988858 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad012c0-a9b6-4ca9-9ebe-b2d505633b2c None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:44.988991 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad012c0-a9b6-4ca9-9ebe-b2d505633b2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:45.006902 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-7ad012c0-a9b6-4ca9-9ebe-b2d505633b2c None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:49:45.008087 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-7ad012c0-a9b6-4ca9-9ebe-b2d505633b2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:49:45.059876 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:45.073589 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 597/1187] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:49:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:49:45.342232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f528b0-0cef-4a97-8cfe-f2e33ee0ab4e None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:45.443173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f528b0-0cef-4a97-8cfe-f2e33ee0ab4e None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:45.460359 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:45.481997 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6ebae2-e784-49a7-af0c-67d757568209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:45.482632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6ebae2-e784-49a7-af0c-67d757568209 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:45.482905 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6ebae2-e784-49a7-af0c-67d757568209 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:45.483173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6ebae2-e784-49a7-af0c-67d757568209 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:45.492779 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6ebae2-e784-49a7-af0c-67d757568209 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:49:45.498044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-ca6ebae2-e784-49a7-af0c-67d757568209 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:49:45.498044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:49:45.498044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:49:45.498044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:49:45.498044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:49:45.498044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:49:45.498044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:49:45.498044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:49:45.498044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:49:45.498044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:49:45.498044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:49:45.498044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:49:45.498044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:49:45.498044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:49:45.498044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:49:45.498044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:49:45.499182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:49:45.499182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:49:45.499182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:49:45.499182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:49:45.499182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:49:45.499182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:49:45.499182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:49:45.499182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:49:45.499182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:49:45.499182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:49:45.499182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:49:45.499182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:49:45.499182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:49:45.499182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:49:45.499182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:49:45.499182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:49:45.500461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:49:45.500461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:49:45.500461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:49:45.500461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:49:45.500461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:49:45.500461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:49:45.500461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:49:45.500461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:49:45.500461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:49:45.500461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:49:45.500461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:49:45.500461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:49:45.500461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 591/1188] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:49:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:49:45.505166 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad012c0-a9b6-4ca9-9ebe-b2d505633b2c None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:45.602337 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad012c0-a9b6-4ca9-9ebe-b2d505633b2c None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:45.636019 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31080c61-5079-43f4-93a9-394226504fa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:45.636629 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31080c61-5079-43f4-93a9-394226504fa5 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:45.636911 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31080c61-5079-43f4-93a9-394226504fa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:45.637185 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31080c61-5079-43f4-93a9-394226504fa5 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:45.637853 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31080c61-5079-43f4-93a9-394226504fa5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:49:45.638179 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31080c61-5079-43f4-93a9-394226504fa5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:45.639559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31080c61-5079-43f4-93a9-394226504fa5 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:45.647634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 598/1189] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:49:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:45.653614 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6ebae2-e784-49a7-af0c-67d757568209 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:45.750608 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6ebae2-e784-49a7-af0c-67d757568209 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:45.783522 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31925795-52af-4124-ac10-2b0028b6b938 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:45.784135 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31925795-52af-4124-ac10-2b0028b6b938 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:45.784411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31925795-52af-4124-ac10-2b0028b6b938 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:45.784682 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31925795-52af-4124-ac10-2b0028b6b938 None admin] PATH_INFO: `/v3/users/083360f37e104e8e8ebd4fa02ea214ad` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:45.785798 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31925795-52af-4124-ac10-2b0028b6b938 None admin] RBAC: Authorizing `identity:get_user(user_id=083360f37e104e8e8ebd4fa02ea214ad)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:45.787243 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31925795-52af-4124-ac10-2b0028b6b938 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:45.788793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 592/1190] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:49:45 2022] GET /identity/v3/users/083360f37e104e8e8ebd4fa02ea214ad => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:45.795446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31080c61-5079-43f4-93a9-394226504fa5 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:45.843804 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:45.892820 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31080c61-5079-43f4-93a9-394226504fa5 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:45.925206 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680a2d78-3b7b-4327-a290-fc8f60d03902 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:45.925834 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680a2d78-3b7b-4327-a290-fc8f60d03902 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:45.926133 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680a2d78-3b7b-4327-a290-fc8f60d03902 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:45.926419 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680a2d78-3b7b-4327-a290-fc8f60d03902 None admin] PATH_INFO: `/v3/projects/882877ae7aca430d91b9e4d0f2f015c0` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:45.927529 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680a2d78-3b7b-4327-a290-fc8f60d03902 None admin] RBAC: Authorizing `identity:get_project(project_id=882877ae7aca430d91b9e4d0f2f015c0)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:45.928966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680a2d78-3b7b-4327-a290-fc8f60d03902 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:45.930642 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 599/1191] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:49:45 2022] GET /identity/v3/projects/882877ae7aca430d91b9e4d0f2f015c0 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:45.937049 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31925795-52af-4124-ac10-2b0028b6b938 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:45.980005 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:46.035381 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31925795-52af-4124-ac10-2b0028b6b938 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:46.069400 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0be0bc-ca5d-4114-aa11-3cd2dcff4764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:46.070044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0be0bc-ca5d-4114-aa11-3cd2dcff4764 None admin] REQUEST_METHOD: `PUT` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:46.070351 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0be0bc-ca5d-4114-aa11-3cd2dcff4764 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:46.070622 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0be0bc-ca5d-4114-aa11-3cd2dcff4764 None admin] PATH_INFO: `/v3/projects/882877ae7aca430d91b9e4d0f2f015c0/users/083360f37e104e8e8ebd4fa02ea214ad/roles/9d5b19a61c46475d8048b1bd66ca25a1` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:46.072507 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0be0bc-ca5d-4114-aa11-3cd2dcff4764 None admin] RBAC: Authorizing `identity:create_grant(project_id=882877ae7aca430d91b9e4d0f2f015c0, user_id=083360f37e104e8e8ebd4fa02ea214ad, role_id=9d5b19a61c46475d8048b1bd66ca25a1)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:46.074323 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0be0bc-ca5d-4114-aa11-3cd2dcff4764 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:46.087012 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 593/1192] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:49:45 2022] PUT /identity/v3/projects/882877ae7aca430d91b9e4d0f2f015c0/users/083360f37e104e8e8ebd4fa02ea214ad/roles/9d5b19a61c46475d8048b1bd66ca25a1 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:49:48.187882 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78714e9-cc95-4ecd-aa94-e3fc47b6b8d6 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:48.188415 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78714e9-cc95-4ecd-aa94-e3fc47b6b8d6 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:48.188415 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78714e9-cc95-4ecd-aa94-e3fc47b6b8d6 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:48.189769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 600/1193] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:49:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:49:48.194923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba597542-17dd-4b82-9dfa-37febd8fe737 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:48.195299 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba597542-17dd-4b82-9dfa-37febd8fe737 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:48.195479 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba597542-17dd-4b82-9dfa-37febd8fe737 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:48.214211 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-ba597542-17dd-4b82-9dfa-37febd8fe737 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:49:48.215402 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-ba597542-17dd-4b82-9dfa-37febd8fe737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:49:48.281320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 594/1194] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:49:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:49:48.554614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78714e9-cc95-4ecd-aa94-e3fc47b6b8d6 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:48.655668 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78714e9-cc95-4ecd-aa94-e3fc47b6b8d6 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:48.673189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:48.690772 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f89021-8037-404d-be85-035fc1a6e8b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:48.691399 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f89021-8037-404d-be85-035fc1a6e8b9 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:48.691677 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f89021-8037-404d-be85-035fc1a6e8b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:48.691948 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f89021-8037-404d-be85-035fc1a6e8b9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:48.702284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f89021-8037-404d-be85-035fc1a6e8b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:49:48.707377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-b2f89021-8037-404d-be85-035fc1a6e8b9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:49:48.707377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:49:48.707377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:49:48.707377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:49:48.707377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:49:48.707377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:49:48.707377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:49:48.707377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:49:48.707377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:49:48.707377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:49:48.707377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:49:48.707377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:49:48.707377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:49:48.707377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:49:48.707377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:49:48.707377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:49:48.708456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:49:48.708456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:49:48.708456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:49:48.708456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:49:48.708456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:49:48.708456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:49:48.708456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:49:48.708456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:49:48.708456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:49:48.708456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:49:48.708456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:49:48.708456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:49:48.708456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:49:48.708456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:49:48.708456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:49:48.708456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:49:48.709510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:49:48.709510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:49:48.709510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:49:48.709510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:49:48.709510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:49:48.709510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:49:48.709510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:49:48.709510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:49:48.709510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:49:48.709510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:49:48.709510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:49:48.709510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:49:48.709510 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 601/1195] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:49:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:49:48.714833 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba597542-17dd-4b82-9dfa-37febd8fe737 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:48.743735 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:48.811080 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba597542-17dd-4b82-9dfa-37febd8fe737 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:48.844213 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3e7cf3-a6cf-4af9-b6dd-7af63c7b8e2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:48.844813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e7cf3-a6cf-4af9-b6dd-7af63c7b8e2a None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:48.845096 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e7cf3-a6cf-4af9-b6dd-7af63c7b8e2a None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:48.845370 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e7cf3-a6cf-4af9-b6dd-7af63c7b8e2a None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:48.846030 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3e7cf3-a6cf-4af9-b6dd-7af63c7b8e2a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:49:48.846358 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3e7cf3-a6cf-4af9-b6dd-7af63c7b8e2a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:48.847711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3e7cf3-a6cf-4af9-b6dd-7af63c7b8e2a None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:48.855520 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 595/1196] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:49:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:49:48.861419 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f89021-8037-404d-be85-035fc1a6e8b9 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:48.959181 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f89021-8037-404d-be85-035fc1a6e8b9 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:48.992157 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f5649f-a139-4320-842f-8fc64d12a8e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:48.992772 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f5649f-a139-4320-842f-8fc64d12a8e3 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:48.993049 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f5649f-a139-4320-842f-8fc64d12a8e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:48.993318 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f5649f-a139-4320-842f-8fc64d12a8e3 None admin] PATH_INFO: `/v3/users/083360f37e104e8e8ebd4fa02ea214ad` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:48.995147 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f5649f-a139-4320-842f-8fc64d12a8e3 None admin] RBAC: Authorizing `identity:get_user(user_id=083360f37e104e8e8ebd4fa02ea214ad)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:48.996570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f5649f-a139-4320-842f-8fc64d12a8e3 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:48.998134 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 602/1197] 10.222.0.27 () {56 vars in 1177 bytes} [Wed Sep 7 03:49:48 2022] GET /identity/v3/users/083360f37e104e8e8ebd4fa02ea214ad => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:49.003852 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3e7cf3-a6cf-4af9-b6dd-7af63c7b8e2a None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:49.095333 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:49.102281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3e7cf3-a6cf-4af9-b6dd-7af63c7b8e2a None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:49.134507 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c675c937-e8ea-4ff1-9116-7b39d73618ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:49.135754 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c675c937-e8ea-4ff1-9116-7b39d73618ad None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:49.136209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c675c937-e8ea-4ff1-9116-7b39d73618ad None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:49.136484 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c675c937-e8ea-4ff1-9116-7b39d73618ad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:49.201207 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c675c937-e8ea-4ff1-9116-7b39d73618ad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:49.202672 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c675c937-e8ea-4ff1-9116-7b39d73618ad None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:49.290445 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:49.298790 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 596/1198] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:49:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:49:49.304722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f5649f-a139-4320-842f-8fc64d12a8e3 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:49.350297 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:49.401264 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f5649f-a139-4320-842f-8fc64d12a8e3 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:49.434597 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6357a3-b2d1-4547-b1e3-84fea99ea00a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:49.435212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6357a3-b2d1-4547-b1e3-84fea99ea00a None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:49.435489 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6357a3-b2d1-4547-b1e3-84fea99ea00a None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:49.435760 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6357a3-b2d1-4547-b1e3-84fea99ea00a None admin] PATH_INFO: `/v3/projects/882877ae7aca430d91b9e4d0f2f015c0` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:49.436860 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6357a3-b2d1-4547-b1e3-84fea99ea00a None admin] RBAC: Authorizing `identity:get_project(project_id=882877ae7aca430d91b9e4d0f2f015c0)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:49.438312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6357a3-b2d1-4547-b1e3-84fea99ea00a None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:49.439921 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 603/1199] 10.222.0.27 () {56 vars in 1186 bytes} [Wed Sep 7 03:49:49 2022] GET /identity/v3/projects/882877ae7aca430d91b9e4d0f2f015c0 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:49.446167 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c675c937-e8ea-4ff1-9116-7b39d73618ad None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:49:49.542577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c675c937-e8ea-4ff1-9116-7b39d73618ad None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:49:49.574585 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e4dc8a-18ff-4d27-9b82-72db1983afe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:49:49.575177 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e4dc8a-18ff-4d27-9b82-72db1983afe3 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:49:49.575454 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e4dc8a-18ff-4d27-9b82-72db1983afe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:49:49.575725 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e4dc8a-18ff-4d27-9b82-72db1983afe3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:49:49.576434 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e4dc8a-18ff-4d27-9b82-72db1983afe3 None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=882877ae7aca430d91b9e4d0f2f015c0, user.id=083360f37e104e8e8ebd4fa02ea214ad) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:49:49.576738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e4dc8a-18ff-4d27-9b82-72db1983afe3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:49:49.578155 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e4dc8a-18ff-4d27-9b82-72db1983afe3 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:49:49.583914 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 597/1200] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:49:49 2022] GET /identity/v3/role_assignments?user.id=083360f37e104e8e8ebd4fa02ea214ad&scope.project.id=882877ae7aca430d91b9e4d0f2f015c0&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:49:59.430404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=43 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:49:59.581396 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=71 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:45.044834 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcb2e60-6805-4945-a8be-849b018d8b2d None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:45.045417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcb2e60-6805-4945-a8be-849b018d8b2d None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:45.045417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcb2e60-6805-4945-a8be-849b018d8b2d None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:45.047468 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 604/1201] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:50:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:50:45.056524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e8e4e2-e6aa-4288-a1a4-a7ab7631a85f None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:45.057281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e8e4e2-e6aa-4288-a1a4-a7ab7631a85f None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:45.057281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e8e4e2-e6aa-4288-a1a4-a7ab7631a85f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:45.079455 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-07e8e4e2-e6aa-4288-a1a4-a7ab7631a85f None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:50:45.081023 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-07e8e4e2-e6aa-4288-a1a4-a7ab7631a85f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:50:45.172821 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 598/1202] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:50:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:50:45.483903 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcb2e60-6805-4945-a8be-849b018d8b2d None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:45.617497 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcb2e60-6805-4945-a8be-849b018d8b2d None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:45.666350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9987db3-40da-447a-9d6e-96fa786c827e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:45.666996 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9987db3-40da-447a-9d6e-96fa786c827e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:45.667278 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9987db3-40da-447a-9d6e-96fa786c827e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:45.667552 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9987db3-40da-447a-9d6e-96fa786c827e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:45.678902 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9987db3-40da-447a-9d6e-96fa786c827e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:45.680576 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9987db3-40da-447a-9d6e-96fa786c827e None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:45.697102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-b9987db3-40da-447a-9d6e-96fa786c827e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:50:45.697102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:50:45.697102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:50:45.697102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:50:45.697102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:50:45.697102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:50:45.697102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:50:45.697102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:50:45.697102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:50:45.697102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:50:45.697102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:50:45.697102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:50:45.697102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:50:45.697102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:50:45.697102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:50:45.697102 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:50:45.698277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:50:45.704923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:50:45.704923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:50:45.704923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:50:45.704923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 605/1203] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:50:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:50:45.705446 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e8e4e2-e6aa-4288-a1a4-a7ab7631a85f None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:45.841175 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e8e4e2-e6aa-4288-a1a4-a7ab7631a85f None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:45.879178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:45.894562 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364b5d12-0f35-4fe6-85e9-fbeb9f96a8bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:45.895260 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364b5d12-0f35-4fe6-85e9-fbeb9f96a8bd None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:45.895550 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364b5d12-0f35-4fe6-85e9-fbeb9f96a8bd None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:45.895868 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364b5d12-0f35-4fe6-85e9-fbeb9f96a8bd None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:45.896565 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364b5d12-0f35-4fe6-85e9-fbeb9f96a8bd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:50:45.896877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364b5d12-0f35-4fe6-85e9-fbeb9f96a8bd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:45.899174 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364b5d12-0f35-4fe6-85e9-fbeb9f96a8bd None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:45.914770 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 599/1204] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:50:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:50:45.924308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9987db3-40da-447a-9d6e-96fa786c827e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:45.996110 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:46.063044 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9987db3-40da-447a-9d6e-96fa786c827e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:46.111020 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecddc8c-2366-4db3-9cd4-a931664dc618 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:46.112144 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecddc8c-2366-4db3-9cd4-a931664dc618 None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:46.112711 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecddc8c-2366-4db3-9cd4-a931664dc618 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:46.113187 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecddc8c-2366-4db3-9cd4-a931664dc618 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:46.114056 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecddc8c-2366-4db3-9cd4-a931664dc618 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:46.115772 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecddc8c-2366-4db3-9cd4-a931664dc618 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:46.153470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 606/1205] 10.222.0.27 () {60 vars in 1132 bytes} [Wed Sep 7 03:50:45 2022] POST /identity/v3/users => generated 251 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:50:48.712608 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6478e7bb-23a6-4cef-bbee-807476d9cb5d None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:48.714304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6478e7bb-23a6-4cef-bbee-807476d9cb5d None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:48.714304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6478e7bb-23a6-4cef-bbee-807476d9cb5d None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:48.714849 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 600/1206] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:50:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:50:48.721074 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c829e599-6476-478e-a603-05bb595fff3a None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:48.722193 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c829e599-6476-478e-a603-05bb595fff3a None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:48.722193 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c829e599-6476-478e-a603-05bb595fff3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:48.741008 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-c829e599-6476-478e-a603-05bb595fff3a None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:50:48.743333 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-c829e599-6476-478e-a603-05bb595fff3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:50:48.825797 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 607/1207] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:50:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:50:49.099882 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6478e7bb-23a6-4cef-bbee-807476d9cb5d None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:49.222921 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6478e7bb-23a6-4cef-bbee-807476d9cb5d None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:49.275788 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f8b083-a929-497b-a4dc-7e793d9eeb65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:49.276532 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f8b083-a929-497b-a4dc-7e793d9eeb65 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:49.276814 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f8b083-a929-497b-a4dc-7e793d9eeb65 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:49.277089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f8b083-a929-497b-a4dc-7e793d9eeb65 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:49.292353 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f8b083-a929-497b-a4dc-7e793d9eeb65 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:50:49.298090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-14f8b083-a929-497b-a4dc-7e793d9eeb65 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:50:49.298090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:50:49.298090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:50:49.298090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:50:49.298090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:50:49.298090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:50:49.298090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:50:49.298090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:50:49.298090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:50:49.298090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:50:49.298090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:50:49.298090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:50:49.298090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:50:49.298090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:50:49.298090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:50:49.298090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:50:49.299302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:50:49.299302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:50:49.299302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:50:49.299302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:50:49.299302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:50:49.299302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:50:49.299302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:50:49.299302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:50:49.299302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:50:49.299302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:50:49.299302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:50:49.299302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:50:49.299302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:50:49.299302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:50:49.299302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:50:49.299302 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:50:49.300309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:50:49.300309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:50:49.300309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:50:49.300309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:50:49.300309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:50:49.300309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:50:49.300309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:50:49.300309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:50:49.300309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:50:49.300309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:50:49.300309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:50:49.300309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:50:49.300309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 601/1208] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:50:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:50:49.307016 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c829e599-6476-478e-a603-05bb595fff3a None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:49.364291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:49.437193 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c829e599-6476-478e-a603-05bb595fff3a None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:49.474554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7082d834-1d9b-46e0-83e8-8f4943461164 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:49.475199 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7082d834-1d9b-46e0-83e8-8f4943461164 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:49.475472 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7082d834-1d9b-46e0-83e8-8f4943461164 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:49.475769 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7082d834-1d9b-46e0-83e8-8f4943461164 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:49.476490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7082d834-1d9b-46e0-83e8-8f4943461164 None admin] RBAC: Adding query filter params (name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:50:49.476800 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7082d834-1d9b-46e0-83e8-8f4943461164 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:49.478882 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7082d834-1d9b-46e0-83e8-8f4943461164 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:49.492578 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 608/1209] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:50:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:50:49.494503 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f8b083-a929-497b-a4dc-7e793d9eeb65 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:49.557260 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:49.627521 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f8b083-a929-497b-a4dc-7e793d9eeb65 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:49.670851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc607d1-fecb-45ab-9358-adf3e5b9dfb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:49.671489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc607d1-fecb-45ab-9358-adf3e5b9dfb8 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:49.671776 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc607d1-fecb-45ab-9358-adf3e5b9dfb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:49.672052 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc607d1-fecb-45ab-9358-adf3e5b9dfb8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:49.682710 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc607d1-fecb-45ab-9358-adf3e5b9dfb8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:49.684220 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc607d1-fecb-45ab-9358-adf3e5b9dfb8 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:49.701050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-ffc607d1-fecb-45ab-9358-adf3e5b9dfb8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:50:49.701050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:50:49.701050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:50:49.701050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:50:49.701050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:50:49.701050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:50:49.701050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:50:49.701050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:50:49.701050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:50:49.701050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:50:49.701050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:50:49.701050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:50:49.701050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:50:49.701050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:50:49.701050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:50:49.701050 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:50:49.702032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:50:49.706916 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:50:49.706916 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:50:49.706916 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:50:49.706916 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 602/1210] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:50:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:50:49.709550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7082d834-1d9b-46e0-83e8-8f4943461164 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:49.831593 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:49.835026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7082d834-1d9b-46e0-83e8-8f4943461164 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:49.894793 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d66b847-cfc9-4ad6-975e-aa244682ac12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:49.895900 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d66b847-cfc9-4ad6-975e-aa244682ac12 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:49.896280 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d66b847-cfc9-4ad6-975e-aa244682ac12 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:49.896452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d66b847-cfc9-4ad6-975e-aa244682ac12 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:49.897111 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d66b847-cfc9-4ad6-975e-aa244682ac12 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:50:49.897439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d66b847-cfc9-4ad6-975e-aa244682ac12 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:49.898826 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d66b847-cfc9-4ad6-975e-aa244682ac12 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:49.912491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 609/1211] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:50:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:50:49.918992 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc607d1-fecb-45ab-9358-adf3e5b9dfb8 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:50.029299 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc607d1-fecb-45ab-9358-adf3e5b9dfb8 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:50.030896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:50.065393 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923bb06f-3f7e-4fbb-9475-7c4889b1149c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:50.066034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923bb06f-3f7e-4fbb-9475-7c4889b1149c None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:50.066337 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923bb06f-3f7e-4fbb-9475-7c4889b1149c None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:50.066614 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923bb06f-3f7e-4fbb-9475-7c4889b1149c None admin] PATH_INFO: `/v3/users/placement` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:50.074691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923bb06f-3f7e-4fbb-9475-7c4889b1149c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:50.076138 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923bb06f-3f7e-4fbb-9475-7c4889b1149c None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:50.088378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-923bb06f-3f7e-4fbb-9475-7c4889b1149c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 07 03:50:50.088378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:50:50.088378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:50:50.088378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:50:50.088378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:50:50.088378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:50:50.088378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:50:50.088378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:50:50.088378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:50:50.088378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:50:50.088378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:50:50.088378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:50:50.088378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:50:50.088378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:50:50.088378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:50:50.088378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:50:50.089378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:50:50.091069 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:50:50.091069 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:50:50.091069 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:50:50.091069 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:50:50.091069 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:50:50.091069 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:50:50.091069 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:50:50.091069 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:50:50.091069 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 07 03:50:50.091069 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:50:50.091069 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 603/1212] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:50:49 2022] GET /identity/v3/users/placement => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:50:50.095909 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d66b847-cfc9-4ad6-975e-aa244682ac12 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:50.212610 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d66b847-cfc9-4ad6-975e-aa244682ac12 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:50.262366 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3a1bdd-ce73-4016-b15a-2dbb17d5d753 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:50.263064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3a1bdd-ce73-4016-b15a-2dbb17d5d753 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:50.263419 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3a1bdd-ce73-4016-b15a-2dbb17d5d753 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:50.263812 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3a1bdd-ce73-4016-b15a-2dbb17d5d753 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:50.264600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3a1bdd-ce73-4016-b15a-2dbb17d5d753 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:50:50.264998 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3a1bdd-ce73-4016-b15a-2dbb17d5d753 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:50.266542 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3a1bdd-ce73-4016-b15a-2dbb17d5d753 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:50.292052 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 610/1213] 10.222.0.27 () {56 vars in 1143 bytes} [Wed Sep 7 03:50:50 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:50:50.295013 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923bb06f-3f7e-4fbb-9475-7c4889b1149c None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:50.407241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923bb06f-3f7e-4fbb-9475-7c4889b1149c None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:50.454392 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5851adaa-88cb-4ffe-9eb4-eb3b7f6311bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:50.455370 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5851adaa-88cb-4ffe-9eb4-eb3b7f6311bf None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:50.455655 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5851adaa-88cb-4ffe-9eb4-eb3b7f6311bf None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:50.455923 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5851adaa-88cb-4ffe-9eb4-eb3b7f6311bf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:50.490750 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:50.533204 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5851adaa-88cb-4ffe-9eb4-eb3b7f6311bf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:50.534894 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5851adaa-88cb-4ffe-9eb4-eb3b7f6311bf None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:50.647051 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 604/1214] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:50:50 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:50:50.654012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3a1bdd-ce73-4016-b15a-2dbb17d5d753 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:50.678234 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:50.779415 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3a1bdd-ce73-4016-b15a-2dbb17d5d753 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:50.823124 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ebe923-2579-4b0f-88e0-36fc70f4e286 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:50.823970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ebe923-2579-4b0f-88e0-36fc70f4e286 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:50.824305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ebe923-2579-4b0f-88e0-36fc70f4e286 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:50.824643 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ebe923-2579-4b0f-88e0-36fc70f4e286 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:50.839575 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ebe923-2579-4b0f-88e0-36fc70f4e286 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:50.841251 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ebe923-2579-4b0f-88e0-36fc70f4e286 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:50.865923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-04ebe923-2579-4b0f-88e0-36fc70f4e286 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:50:50.865923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:50:50.865923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:50:50.865923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:50:50.865923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:50:50.865923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:50:50.865923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:50:50.865923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:50:50.865923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:50:50.865923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:50:50.865923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:50:50.865923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:50:50.865923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:50:50.865923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:50:50.865923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:50:50.865923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:50:50.867454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:50:50.869207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:50:50.869207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:50:50.869207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:50:50.869207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 611/1215] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:50:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:50:50.875256 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5851adaa-88cb-4ffe-9eb4-eb3b7f6311bf None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:50.955702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:51.001199 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5851adaa-88cb-4ffe-9eb4-eb3b7f6311bf None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:51.044403 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bbcca0-cf81-4572-9294-f85b3f4b8583 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:51.044959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bbcca0-cf81-4572-9294-f85b3f4b8583 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:51.045236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bbcca0-cf81-4572-9294-f85b3f4b8583 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:51.045561 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bbcca0-cf81-4572-9294-f85b3f4b8583 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:51.046316 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bbcca0-cf81-4572-9294-f85b3f4b8583 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:50:51.046735 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bbcca0-cf81-4572-9294-f85b3f4b8583 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:51.048932 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bbcca0-cf81-4572-9294-f85b3f4b8583 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:51.064231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 605/1216] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:50:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:50:51.070032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ebe923-2579-4b0f-88e0-36fc70f4e286 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:51.149313 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:51.183710 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ebe923-2579-4b0f-88e0-36fc70f4e286 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:51.219825 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8661885-d9c5-4fc8-9bcf-718b3d8b4c94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:51.220464 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8661885-d9c5-4fc8-9bcf-718b3d8b4c94 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:51.220739 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8661885-d9c5-4fc8-9bcf-718b3d8b4c94 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:51.221009 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8661885-d9c5-4fc8-9bcf-718b3d8b4c94 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:51.228714 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8661885-d9c5-4fc8-9bcf-718b3d8b4c94 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:51.230319 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8661885-d9c5-4fc8-9bcf-718b3d8b4c94 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:51.243357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-b8661885-d9c5-4fc8-9bcf-718b3d8b4c94 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:50:51.243357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:50:51.243357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:50:51.243357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:50:51.243357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:50:51.243357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:50:51.243357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:50:51.243357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:50:51.243357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:50:51.243357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:50:51.243357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:50:51.243357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:50:51.243357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:50:51.243357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:50:51.243357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:50:51.244580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:50:51.245735 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:50:51.245735 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:50:51.245735 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:50:51.245735 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:50:51.245735 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:50:51.245735 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:50:51.245735 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:50:51.245735 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:50:51.245735 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 612/1217] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:50:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:50:51.251237 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bbcca0-cf81-4572-9294-f85b3f4b8583 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:51.363514 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bbcca0-cf81-4572-9294-f85b3f4b8583 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:51.391468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:51.406009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2f5ba8-7e7f-4131-a233-3226e9dff57d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:51.406658 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2f5ba8-7e7f-4131-a233-3226e9dff57d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:51.406954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2f5ba8-7e7f-4131-a233-3226e9dff57d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:51.407211 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2f5ba8-7e7f-4131-a233-3226e9dff57d None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:51.407903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2f5ba8-7e7f-4131-a233-3226e9dff57d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:50:51.408232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2f5ba8-7e7f-4131-a233-3226e9dff57d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:51.409658 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2f5ba8-7e7f-4131-a233-3226e9dff57d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:51.421584 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 606/1218] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:50:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:50:51.427622 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8661885-d9c5-4fc8-9bcf-718b3d8b4c94 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:51.540528 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8661885-d9c5-4fc8-9bcf-718b3d8b4c94 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:51.570292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:51.577824 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abaabd9-f10d-441e-8d39-c6d08f9c1553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:51.578482 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abaabd9-f10d-441e-8d39-c6d08f9c1553 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:51.578832 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abaabd9-f10d-441e-8d39-c6d08f9c1553 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:51.579122 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abaabd9-f10d-441e-8d39-c6d08f9c1553 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:51.579833 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abaabd9-f10d-441e-8d39-c6d08f9c1553 None admin] RBAC: Adding query filter params (role.id=a7b16bb097f14f78a7a31c04bd6d32bd, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=3bc3872e11a749758665f2c5dc1db8e9) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:50:51.580392 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abaabd9-f10d-441e-8d39-c6d08f9c1553 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:51.581859 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abaabd9-f10d-441e-8d39-c6d08f9c1553 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:51.588777 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 613/1219] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:50:51 2022] GET /identity/v3/role_assignments?user.id=3bc3872e11a749758665f2c5dc1db8e9&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=a7b16bb097f14f78a7a31c04bd6d32bd => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:50:53.959666 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4356218b-bb62-4e66-9df8-85a1bfe7430c None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:53.960314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4356218b-bb62-4e66-9df8-85a1bfe7430c None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:53.960314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4356218b-bb62-4e66-9df8-85a1bfe7430c None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:53.961974 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 607/1220] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:50:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:50:53.968266 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830562bb-929d-4059-9819-e70a40b5c70a None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:53.968744 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830562bb-929d-4059-9819-e70a40b5c70a None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:53.968984 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830562bb-929d-4059-9819-e70a40b5c70a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:53.988975 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-830562bb-929d-4059-9819-e70a40b5c70a None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:50:53.990203 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-830562bb-929d-4059-9819-e70a40b5c70a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:50:54.071782 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 614/1221] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:50:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:50:54.370701 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4356218b-bb62-4e66-9df8-85a1bfe7430c None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:54.483645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4356218b-bb62-4e66-9df8-85a1bfe7430c None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:54.522380 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d42869-b296-4623-9825-27796786fcf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:54.522984 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d42869-b296-4623-9825-27796786fcf2 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:54.523232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d42869-b296-4623-9825-27796786fcf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:54.523509 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d42869-b296-4623-9825-27796786fcf2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:54.534789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d42869-b296-4623-9825-27796786fcf2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:50:54.540986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-65d42869-b296-4623-9825-27796786fcf2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:50:54.540986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:50:54.540986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:50:54.540986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:50:54.540986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:50:54.540986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:50:54.540986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:50:54.540986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:50:54.540986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:50:54.540986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:50:54.540986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:50:54.540986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:50:54.540986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:50:54.540986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:50:54.540986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:50:54.540986 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:50:54.546541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:50:54.546541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:50:54.546541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:50:54.546541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:50:54.546541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:50:54.546541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:50:54.546541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:50:54.546541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:50:54.546541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:50:54.546541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:50:54.546541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:50:54.546541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:50:54.546541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:50:54.546541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:50:54.546541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:50:54.546541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:50:54.547574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:50:54.547574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:50:54.547574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:50:54.547574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:50:54.547574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:50:54.547574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:50:54.547574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:50:54.547574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:50:54.547574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:50:54.547574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:50:54.547574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:50:54.547574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:50:54.547574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 608/1222] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:50:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:50:54.548747 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830562bb-929d-4059-9819-e70a40b5c70a None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:54.672537 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830562bb-929d-4059-9819-e70a40b5c70a None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:54.701934 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:54.715218 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c77528-2616-4a61-a095-8c8646f8d33a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:54.716287 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c77528-2616-4a61-a095-8c8646f8d33a None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:54.716571 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c77528-2616-4a61-a095-8c8646f8d33a None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:54.716966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c77528-2616-4a61-a095-8c8646f8d33a None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:54.717646 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c77528-2616-4a61-a095-8c8646f8d33a None admin] RBAC: Adding query filter params (name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:50:54.717952 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c77528-2616-4a61-a095-8c8646f8d33a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:54.719378 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c77528-2616-4a61-a095-8c8646f8d33a None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:54.730086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 615/1223] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:50:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:50:54.736137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d42869-b296-4623-9825-27796786fcf2 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:54.785436 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:54.849534 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d42869-b296-4623-9825-27796786fcf2 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:54.888634 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b4715d-b9a3-4dc1-a928-826746dee167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:54.889364 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b4715d-b9a3-4dc1-a928-826746dee167 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:54.889645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b4715d-b9a3-4dc1-a928-826746dee167 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:54.890073 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b4715d-b9a3-4dc1-a928-826746dee167 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:54.901285 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b4715d-b9a3-4dc1-a928-826746dee167 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:54.902764 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b4715d-b9a3-4dc1-a928-826746dee167 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:54.920129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-b5b4715d-b9a3-4dc1-a928-826746dee167 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:50:54.920129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:50:54.920129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:50:54.920129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:50:54.920129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:50:54.920129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:50:54.920129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:50:54.920129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:50:54.920129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:50:54.920129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:50:54.920129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:50:54.920129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:50:54.920129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:50:54.920129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:50:54.920129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:50:54.920129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:50:54.921231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:50:54.922199 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:50:54.922199 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:50:54.922199 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:50:54.922199 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 609/1224] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:50:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:50:54.927516 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c77528-2616-4a61-a095-8c8646f8d33a None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:55.041007 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c77528-2616-4a61-a095-8c8646f8d33a None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:55.081747 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23c6bae-274f-4cb5-9db0-9b6be890fa62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:55.082409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23c6bae-274f-4cb5-9db0-9b6be890fa62 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:55.082930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23c6bae-274f-4cb5-9db0-9b6be890fa62 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:55.083111 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23c6bae-274f-4cb5-9db0-9b6be890fa62 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:55.083797 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23c6bae-274f-4cb5-9db0-9b6be890fa62 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:50:55.084281 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23c6bae-274f-4cb5-9db0-9b6be890fa62 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:55.085697 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23c6bae-274f-4cb5-9db0-9b6be890fa62 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:55.099149 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 616/1225] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:50:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:50:55.105382 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b4715d-b9a3-4dc1-a928-826746dee167 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:55.207194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:55.215173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b4715d-b9a3-4dc1-a928-826746dee167 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:55.257468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff8addb-dc06-4137-82cb-b62a7e25fccd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:55.258128 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff8addb-dc06-4137-82cb-b62a7e25fccd None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:55.259027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff8addb-dc06-4137-82cb-b62a7e25fccd None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:55.259305 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff8addb-dc06-4137-82cb-b62a7e25fccd None admin] PATH_INFO: `/v3/users/placement` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:55.269789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff8addb-dc06-4137-82cb-b62a7e25fccd None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:55.271786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff8addb-dc06-4137-82cb-b62a7e25fccd None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:55.286597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-8ff8addb-dc06-4137-82cb-b62a7e25fccd None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 07 03:50:55.286597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:50:55.286597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:50:55.286597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:50:55.286597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:50:55.286597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:50:55.286597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:50:55.286597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:50:55.286597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:50:55.286597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:50:55.286597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:50:55.286597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:50:55.286597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:50:55.286597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:50:55.286597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:50:55.286597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:50:55.293847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:50:55.295035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:50:55.295035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:50:55.295035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:50:55.295035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:50:55.295035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:50:55.295035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:50:55.295035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:50:55.295035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:50:55.295035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 07 03:50:55.295035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:50:55.295035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 610/1226] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:50:55 2022] GET /identity/v3/users/placement => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:50:55.295668 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23c6bae-274f-4cb5-9db0-9b6be890fa62 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:55.347531 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:55.409570 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23c6bae-274f-4cb5-9db0-9b6be890fa62 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:55.444858 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be49f8c0-8269-4724-8985-afa57347ead3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:55.445467 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be49f8c0-8269-4724-8985-afa57347ead3 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:55.445788 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be49f8c0-8269-4724-8985-afa57347ead3 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:55.446627 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be49f8c0-8269-4724-8985-afa57347ead3 None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:55.447325 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be49f8c0-8269-4724-8985-afa57347ead3 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:50:55.447630 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be49f8c0-8269-4724-8985-afa57347ead3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:55.449086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be49f8c0-8269-4724-8985-afa57347ead3 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:55.468064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 617/1227] 10.222.0.27 () {56 vars in 1143 bytes} [Wed Sep 7 03:50:55 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:50:55.474200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff8addb-dc06-4137-82cb-b62a7e25fccd None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:55.583461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff8addb-dc06-4137-82cb-b62a7e25fccd None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:55.625739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25aa0d2a-5cc2-4aa6-8f9f-479120472daa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:55.626463 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25aa0d2a-5cc2-4aa6-8f9f-479120472daa None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:55.626750 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25aa0d2a-5cc2-4aa6-8f9f-479120472daa None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:55.627021 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25aa0d2a-5cc2-4aa6-8f9f-479120472daa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:55.638985 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25aa0d2a-5cc2-4aa6-8f9f-479120472daa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:55.640411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25aa0d2a-5cc2-4aa6-8f9f-479120472daa None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:55.656488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-25aa0d2a-5cc2-4aa6-8f9f-479120472daa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:50:55.656488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:50:55.656488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:50:55.656488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:50:55.656488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:50:55.656488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:50:55.656488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:50:55.656488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:50:55.656488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:50:55.656488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:50:55.656488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:50:55.656488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:50:55.656488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:50:55.656488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:50:55.656488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:50:55.656488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:50:55.657699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:50:55.658786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:50:55.658786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:50:55.658786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:50:55.658786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 611/1228] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:50:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:50:55.663863 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be49f8c0-8269-4724-8985-afa57347ead3 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:55.776835 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be49f8c0-8269-4724-8985-afa57347ead3 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:55.779284 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:55.815717 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc284581-d209-440b-8af9-d16b1600181c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:55.816353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc284581-d209-440b-8af9-d16b1600181c None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:55.816618 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc284581-d209-440b-8af9-d16b1600181c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:55.816896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc284581-d209-440b-8af9-d16b1600181c None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:55.817545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc284581-d209-440b-8af9-d16b1600181c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:50:55.817853 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc284581-d209-440b-8af9-d16b1600181c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:55.819504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc284581-d209-440b-8af9-d16b1600181c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:55.831345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 618/1229] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:50:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:50:55.837221 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25aa0d2a-5cc2-4aa6-8f9f-479120472daa None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:55.847585 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:55.945319 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25aa0d2a-5cc2-4aa6-8f9f-479120472daa None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:55.982900 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a19b53c-bf68-4c24-8f61-49d52acd35c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:55.983528 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a19b53c-bf68-4c24-8f61-49d52acd35c5 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:55.984404 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a19b53c-bf68-4c24-8f61-49d52acd35c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:55.984404 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a19b53c-bf68-4c24-8f61-49d52acd35c5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:55.990992 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a19b53c-bf68-4c24-8f61-49d52acd35c5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:55.992441 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a19b53c-bf68-4c24-8f61-49d52acd35c5 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:56.004281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-7a19b53c-bf68-4c24-8f61-49d52acd35c5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:50:56.004281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:50:56.004281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:50:56.004281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:50:56.004281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:50:56.004281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:50:56.004281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:50:56.004281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:50:56.004281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:50:56.004281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:50:56.004281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:50:56.004281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:50:56.004281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:50:56.004281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:50:56.004281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:50:56.005205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:50:56.008290 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:50:56.008290 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:50:56.008290 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:50:56.008290 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:50:56.008290 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:50:56.008290 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:50:56.008290 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:50:56.008290 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:50:56.010243 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 612/1230] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:50:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:50:56.011589 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc284581-d209-440b-8af9-d16b1600181c None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:56.126586 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc284581-d209-440b-8af9-d16b1600181c None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:56.165600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740552a6-4ffa-4489-8804-201008aff466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:56.166305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740552a6-4ffa-4489-8804-201008aff466 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:56.166589 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740552a6-4ffa-4489-8804-201008aff466 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:56.166864 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740552a6-4ffa-4489-8804-201008aff466 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:56.167557 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740552a6-4ffa-4489-8804-201008aff466 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:50:56.167890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740552a6-4ffa-4489-8804-201008aff466 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:56.169327 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740552a6-4ffa-4489-8804-201008aff466 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:56.183757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 619/1231] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:50:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:50:56.190549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a19b53c-bf68-4c24-8f61-49d52acd35c5 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:56.269259 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:56.305998 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a19b53c-bf68-4c24-8f61-49d52acd35c5 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:56.341019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031505e9-d8bc-416e-811d-b7185aac8a51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:56.341715 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031505e9-d8bc-416e-811d-b7185aac8a51 None admin] REQUEST_METHOD: `PUT` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:56.341990 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031505e9-d8bc-416e-811d-b7185aac8a51 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:56.342300 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031505e9-d8bc-416e-811d-b7185aac8a51 None admin] PATH_INFO: `/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/3bc3872e11a749758665f2c5dc1db8e9/roles/a7b16bb097f14f78a7a31c04bd6d32bd` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:56.356992 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031505e9-d8bc-416e-811d-b7185aac8a51 None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4816e95144485e8135e5b8f68c3662, user_id=3bc3872e11a749758665f2c5dc1db8e9, role_id=a7b16bb097f14f78a7a31c04bd6d32bd)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:56.358633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031505e9-d8bc-416e-811d-b7185aac8a51 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:56.372857 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 613/1232] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:50:56 2022] PUT /identity/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/3bc3872e11a749758665f2c5dc1db8e9/roles/a7b16bb097f14f78a7a31c04bd6d32bd => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:50:58.550252 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe795d31-a753-44ca-947e-e93f76088583 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:58.550944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe795d31-a753-44ca-947e-e93f76088583 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:58.550944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe795d31-a753-44ca-947e-e93f76088583 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:58.551966 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 620/1233] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:50:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:50:58.558954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eab5997-379a-47ea-8386-462a39d8d802 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:58.559428 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eab5997-379a-47ea-8386-462a39d8d802 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:58.559561 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eab5997-379a-47ea-8386-462a39d8d802 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:58.585200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-1eab5997-379a-47ea-8386-462a39d8d802 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:50:58.587604 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-1eab5997-379a-47ea-8386-462a39d8d802 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:50:58.675334 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 614/1234] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:50:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:50:58.980702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe795d31-a753-44ca-947e-e93f76088583 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:59.007920 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:59.093838 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe795d31-a753-44ca-947e-e93f76088583 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:59.136971 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9460624b-c434-4b0c-b9b7-a3397513749e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:59.137811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9460624b-c434-4b0c-b9b7-a3397513749e None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:59.138186 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9460624b-c434-4b0c-b9b7-a3397513749e None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:59.138459 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9460624b-c434-4b0c-b9b7-a3397513749e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:59.151409 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9460624b-c434-4b0c-b9b7-a3397513749e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:50:59.157086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-9460624b-c434-4b0c-b9b7-a3397513749e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:50:59.157086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:50:59.157086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:50:59.157086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:50:59.157086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:50:59.157086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:50:59.157086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:50:59.157086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:50:59.157086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:50:59.157086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:50:59.157086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:50:59.157086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:50:59.157086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:50:59.157086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:50:59.157086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:50:59.157086 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:50:59.159791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:50:59.159791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:50:59.159791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:50:59.159791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:50:59.159791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:50:59.159791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:50:59.159791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:50:59.159791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:50:59.159791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:50:59.159791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:50:59.159791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:50:59.159791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:50:59.159791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:50:59.159791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:50:59.159791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:50:59.159791 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:50:59.160713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:50:59.160713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:50:59.160713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:50:59.160713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:50:59.160713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:50:59.160713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:50:59.160713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:50:59.160713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:50:59.160713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:50:59.160713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:50:59.160713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 07 03:50:59.160713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:50:59.160713 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 621/1235] 10.222.0.27 () {56 vars in 1102 bytes} [Wed Sep 7 03:50:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:50:59.165304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eab5997-379a-47ea-8386-462a39d8d802 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:59.220304 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:59.285538 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eab5997-379a-47ea-8386-462a39d8d802 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:59.325947 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f5b2ad-4008-4ce4-ab67-b8425c8d42d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:59.326775 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f5b2ad-4008-4ce4-ab67-b8425c8d42d9 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:59.327005 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f5b2ad-4008-4ce4-ab67-b8425c8d42d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:59.327286 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f5b2ad-4008-4ce4-ab67-b8425c8d42d9 None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:59.327959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f5b2ad-4008-4ce4-ab67-b8425c8d42d9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:50:59.328267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f5b2ad-4008-4ce4-ab67-b8425c8d42d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:59.329634 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f5b2ad-4008-4ce4-ab67-b8425c8d42d9 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:59.339436 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 615/1236] 10.222.0.27 () {56 vars in 1103 bytes} [Wed Sep 7 03:50:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:50:59.345508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9460624b-c434-4b0c-b9b7-a3397513749e None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:59.439643 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:59.458537 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9460624b-c434-4b0c-b9b7-a3397513749e None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:59.498607 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb65cc4e-8937-4f28-ab04-4bc77d5f3f56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:59.499285 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb65cc4e-8937-4f28-ab04-4bc77d5f3f56 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:59.499630 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb65cc4e-8937-4f28-ab04-4bc77d5f3f56 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:59.499965 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb65cc4e-8937-4f28-ab04-4bc77d5f3f56 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:59.513749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb65cc4e-8937-4f28-ab04-4bc77d5f3f56 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:59.515287 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb65cc4e-8937-4f28-ab04-4bc77d5f3f56 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:59.532024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-eb65cc4e-8937-4f28-ab04-4bc77d5f3f56 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:50:59.532024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:50:59.532024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:50:59.532024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:50:59.532024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:50:59.532024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:50:59.532024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:50:59.532024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:50:59.532024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:50:59.532024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:50:59.532024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:50:59.532024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:50:59.532024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:50:59.532024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:50:59.532024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:50:59.532024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:50:59.533357 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:50:59.534292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:50:59.534292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:50:59.534292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:50:59.534292 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 622/1237] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:50:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:50:59.539596 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f5b2ad-4008-4ce4-ab67-b8425c8d42d9 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:59.656573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f5b2ad-4008-4ce4-ab67-b8425c8d42d9 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:59.666161 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:59.694174 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb8b4d7-f184-44d5-910c-60ba0040fc18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:59.694785 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb8b4d7-f184-44d5-910c-60ba0040fc18 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:59.695062 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb8b4d7-f184-44d5-910c-60ba0040fc18 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:59.695470 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb8b4d7-f184-44d5-910c-60ba0040fc18 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:59.696883 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb8b4d7-f184-44d5-910c-60ba0040fc18 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:50:59.697205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb8b4d7-f184-44d5-910c-60ba0040fc18 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:59.698639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb8b4d7-f184-44d5-910c-60ba0040fc18 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:59.712707 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 616/1238] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:50:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:50:59.719634 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb65cc4e-8937-4f28-ab04-4bc77d5f3f56 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:50:59.824976 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb65cc4e-8937-4f28-ab04-4bc77d5f3f56 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:50:59.860440 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e7ebdf-3496-4e3a-a110-99f3a8b914df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:50:59.861067 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e7ebdf-3496-4e3a-a110-99f3a8b914df None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:50:59.861345 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e7ebdf-3496-4e3a-a110-99f3a8b914df None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:50:59.861614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e7ebdf-3496-4e3a-a110-99f3a8b914df None admin] PATH_INFO: `/v3/users/placement` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:50:59.864391 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:50:59.870458 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e7ebdf-3496-4e3a-a110-99f3a8b914df None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:50:59.872198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e7ebdf-3496-4e3a-a110-99f3a8b914df None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:50:59.884207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-05e7ebdf-3496-4e3a-a110-99f3a8b914df None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 07 03:50:59.884207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:50:59.884207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:50:59.884207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:50:59.884207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:50:59.884207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:50:59.884207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:50:59.884207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:50:59.884207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:50:59.884207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:50:59.884207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:50:59.884207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:50:59.884207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:50:59.884207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:50:59.884207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:50:59.884207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:50:59.885316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:50:59.886307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:50:59.886307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:50:59.886307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:50:59.886307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:50:59.886307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:50:59.886307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:50:59.886307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:50:59.886307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:50:59.886307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 07 03:50:59.886307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:50:59.886307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 623/1239] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:50:59 2022] GET /identity/v3/users/placement => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 07 03:50:59.891516 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb8b4d7-f184-44d5-910c-60ba0040fc18 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:00.005804 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb8b4d7-f184-44d5-910c-60ba0040fc18 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:00.045065 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3877c4fd-4dc6-44db-b0ae-ebcb3e5ad1bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:00.045928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3877c4fd-4dc6-44db-b0ae-ebcb3e5ad1bc None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:00.046342 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3877c4fd-4dc6-44db-b0ae-ebcb3e5ad1bc None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:00.046630 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3877c4fd-4dc6-44db-b0ae-ebcb3e5ad1bc None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:00.047588 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3877c4fd-4dc6-44db-b0ae-ebcb3e5ad1bc None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:00.047901 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3877c4fd-4dc6-44db-b0ae-ebcb3e5ad1bc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:00.049334 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3877c4fd-4dc6-44db-b0ae-ebcb3e5ad1bc None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:00.070132 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 617/1240] 10.222.0.27 () {56 vars in 1143 bytes} [Wed Sep 7 03:50:59 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:51:00.076664 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e7ebdf-3496-4e3a-a110-99f3a8b914df None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:00.189119 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e7ebdf-3496-4e3a-a110-99f3a8b914df None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:00.224621 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58a5cdc-e971-46aa-b6e2-90be822a698d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:00.225212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58a5cdc-e971-46aa-b6e2-90be822a698d None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:00.225499 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58a5cdc-e971-46aa-b6e2-90be822a698d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:00.225775 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58a5cdc-e971-46aa-b6e2-90be822a698d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:00.294879 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:00.301457 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58a5cdc-e971-46aa-b6e2-90be822a698d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:00.302959 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58a5cdc-e971-46aa-b6e2-90be822a698d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:00.407461 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 624/1241] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:51:00 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 07 03:51:00.413490 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3877c4fd-4dc6-44db-b0ae-ebcb3e5ad1bc None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:00.459065 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:00.549699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3877c4fd-4dc6-44db-b0ae-ebcb3e5ad1bc None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:00.597872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0212cf76-7b19-4579-aff1-2e4b45a573db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:00.598972 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0212cf76-7b19-4579-aff1-2e4b45a573db None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:00.599348 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0212cf76-7b19-4579-aff1-2e4b45a573db None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:00.599666 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0212cf76-7b19-4579-aff1-2e4b45a573db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:00.618099 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0212cf76-7b19-4579-aff1-2e4b45a573db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:00.620644 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0212cf76-7b19-4579-aff1-2e4b45a573db None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:00.648452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-0212cf76-7b19-4579-aff1-2e4b45a573db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:51:00.648452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:00.648452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:00.648452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:00.648452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:00.648452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:00.648452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:00.648452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:00.648452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:00.648452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:00.648452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:00.648452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:00.648452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:51:00.648452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:51:00.648452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:51:00.648452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:00.649691 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:51:00.655735 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:51:00.655735 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:51:00.655735 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:51:00.655735 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 618/1242] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:51:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:00.656012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58a5cdc-e971-46aa-b6e2-90be822a698d None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:00.768939 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:00.775447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58a5cdc-e971-46aa-b6e2-90be822a698d None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:00.813162 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bd2f4e-cb04-4778-8cbf-46e776043fc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:00.813846 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bd2f4e-cb04-4778-8cbf-46e776043fc3 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:00.814239 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bd2f4e-cb04-4778-8cbf-46e776043fc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:00.814520 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bd2f4e-cb04-4778-8cbf-46e776043fc3 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:00.815265 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bd2f4e-cb04-4778-8cbf-46e776043fc3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:00.815573 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bd2f4e-cb04-4778-8cbf-46e776043fc3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:00.816935 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bd2f4e-cb04-4778-8cbf-46e776043fc3 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:00.832055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 625/1243] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:51:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:00.838244 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0212cf76-7b19-4579-aff1-2e4b45a573db None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:00.928629 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:00.953508 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0212cf76-7b19-4579-aff1-2e4b45a573db None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:00.991896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d3cab5-e01c-4381-a6cb-1a4ed3435162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:00.992855 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d3cab5-e01c-4381-a6cb-1a4ed3435162 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:00.993136 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d3cab5-e01c-4381-a6cb-1a4ed3435162 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:00.993393 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d3cab5-e01c-4381-a6cb-1a4ed3435162 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:01.000522 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d3cab5-e01c-4381-a6cb-1a4ed3435162 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:01.002348 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d3cab5-e01c-4381-a6cb-1a4ed3435162 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:01.015331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-43d3cab5-e01c-4381-a6cb-1a4ed3435162 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:51:01.015331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:01.015331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:01.015331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:01.015331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:01.015331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:01.015331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:01.015331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:01.015331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:01.015331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:01.015331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:01.015331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:01.015331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:51:01.015331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:51:01.015331 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:01.016639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:01.017739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:51:01.017739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:51:01.017739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:51:01.017739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:51:01.017739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:51:01.017739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:51:01.017739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:51:01.017739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:51:01.017739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 619/1244] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:51:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:01.022549 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bd2f4e-cb04-4778-8cbf-46e776043fc3 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:01.139851 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bd2f4e-cb04-4778-8cbf-46e776043fc3 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:01.178471 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b02b17-e451-4fa1-a304-1f3e7a9458df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:01.179090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b02b17-e451-4fa1-a304-1f3e7a9458df None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:01.179367 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b02b17-e451-4fa1-a304-1f3e7a9458df None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:01.179639 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b02b17-e451-4fa1-a304-1f3e7a9458df None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:01.180371 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b02b17-e451-4fa1-a304-1f3e7a9458df None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:01.180676 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b02b17-e451-4fa1-a304-1f3e7a9458df None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:01.182177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b02b17-e451-4fa1-a304-1f3e7a9458df None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:01.195065 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 626/1245] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:51:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:01.202491 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d3cab5-e01c-4381-a6cb-1a4ed3435162 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:01.318521 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d3cab5-e01c-4381-a6cb-1a4ed3435162 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:01.356003 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfb1d00-1821-4e33-8b32-beded5122a19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:01.356646 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfb1d00-1821-4e33-8b32-beded5122a19 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:01.356962 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfb1d00-1821-4e33-8b32-beded5122a19 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:01.357263 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfb1d00-1821-4e33-8b32-beded5122a19 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:01.358085 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfb1d00-1821-4e33-8b32-beded5122a19 None admin] RBAC: Adding query filter params (role.id=a7b16bb097f14f78a7a31c04bd6d32bd, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=3bc3872e11a749758665f2c5dc1db8e9) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:01.358431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfb1d00-1821-4e33-8b32-beded5122a19 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:01.359946 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfb1d00-1821-4e33-8b32-beded5122a19 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:01.366317 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 620/1246] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:51:01 2022] GET /identity/v3/role_assignments?user.id=3bc3872e11a749758665f2c5dc1db8e9&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=a7b16bb097f14f78a7a31c04bd6d32bd => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:03.744561 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1085279e-9ea2-40b4-96a5-680b71a05742 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:03.745096 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1085279e-9ea2-40b4-96a5-680b71a05742 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:03.745096 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1085279e-9ea2-40b4-96a5-680b71a05742 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:03.746467 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 627/1247] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:51:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:51:03.752128 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870ac47a-4693-41ee-baf2-4fe43a751b8a None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:03.752571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870ac47a-4693-41ee-baf2-4fe43a751b8a None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:03.752810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870ac47a-4693-41ee-baf2-4fe43a751b8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:03.759897 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:03.775991 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-870ac47a-4693-41ee-baf2-4fe43a751b8a None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:51:03.777203 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-870ac47a-4693-41ee-baf2-4fe43a751b8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:51:03.851373 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 621/1248] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:51:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:51:04.146367 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1085279e-9ea2-40b4-96a5-680b71a05742 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:04.165358 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:04.258413 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1085279e-9ea2-40b4-96a5-680b71a05742 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:04.294967 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77464f6a-d8eb-4455-ac43-6eb8de07889b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:04.295654 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77464f6a-d8eb-4455-ac43-6eb8de07889b None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:04.295946 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77464f6a-d8eb-4455-ac43-6eb8de07889b None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:04.296241 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77464f6a-d8eb-4455-ac43-6eb8de07889b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:04.308269 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77464f6a-d8eb-4455-ac43-6eb8de07889b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:51:04.313501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-77464f6a-d8eb-4455-ac43-6eb8de07889b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:51:04.313501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:04.313501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:04.313501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:04.313501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:04.313501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:04.313501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:04.313501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:04.313501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:04.313501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:04.313501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:04.313501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:04.313501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:51:04.313501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:51:04.313501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:51:04.313501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:51:04.314719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:51:04.314719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:51:04.314719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:51:04.314719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:51:04.314719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:04.314719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:04.314719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:04.314719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:04.314719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:04.314719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:04.314719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:04.314719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:04.314719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:04.314719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:04.314719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:04.314719 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:04.319452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:04.319452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:04.319452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:04.319452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:04.319452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:51:04.319452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:51:04.319452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:51:04.319452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:51:04.319452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:51:04.319452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:51:04.319452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:51:04.319452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:51:04.319452 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 628/1249] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:51:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:04.321577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870ac47a-4693-41ee-baf2-4fe43a751b8a None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:04.425579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870ac47a-4693-41ee-baf2-4fe43a751b8a None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:04.457453 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:04.463660 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79253e41-eecd-4f81-a402-43bf9cff2ceb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:04.464292 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79253e41-eecd-4f81-a402-43bf9cff2ceb None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:04.464570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79253e41-eecd-4f81-a402-43bf9cff2ceb None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:04.464839 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79253e41-eecd-4f81-a402-43bf9cff2ceb None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:04.465517 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79253e41-eecd-4f81-a402-43bf9cff2ceb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:04.465816 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79253e41-eecd-4f81-a402-43bf9cff2ceb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:04.467228 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79253e41-eecd-4f81-a402-43bf9cff2ceb None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:04.481549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 622/1250] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:51:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:04.482827 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77464f6a-d8eb-4455-ac43-6eb8de07889b None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:04.563804 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:04.593250 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77464f6a-d8eb-4455-ac43-6eb8de07889b None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:04.639918 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23057e2-9e86-494d-b748-85602d25b124 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:04.640568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23057e2-9e86-494d-b748-85602d25b124 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:04.640855 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23057e2-9e86-494d-b748-85602d25b124 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:04.641131 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23057e2-9e86-494d-b748-85602d25b124 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:04.651950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23057e2-9e86-494d-b748-85602d25b124 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:04.653430 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23057e2-9e86-494d-b748-85602d25b124 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:04.669245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-e23057e2-9e86-494d-b748-85602d25b124 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:51:04.669245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:04.669245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:04.669245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:04.669245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:04.669245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:04.669245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:04.669245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:04.669245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:04.669245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:04.669245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:04.669245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:04.669245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:51:04.669245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:51:04.669245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:51:04.669245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:04.670530 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:51:04.671420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:51:04.671420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:51:04.671420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:51:04.671420 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 629/1251] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:51:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:04.676406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79253e41-eecd-4f81-a402-43bf9cff2ceb None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:04.778888 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79253e41-eecd-4f81-a402-43bf9cff2ceb None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:04.813504 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5b77ea-7914-49fa-806e-0614a0ed980d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:04.814147 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5b77ea-7914-49fa-806e-0614a0ed980d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:04.814438 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5b77ea-7914-49fa-806e-0614a0ed980d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:04.814711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5b77ea-7914-49fa-806e-0614a0ed980d None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:04.815362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5b77ea-7914-49fa-806e-0614a0ed980d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:04.815670 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5b77ea-7914-49fa-806e-0614a0ed980d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:04.817022 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5b77ea-7914-49fa-806e-0614a0ed980d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:04.828972 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 623/1252] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:51:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:04.835094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23057e2-9e86-494d-b748-85602d25b124 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:04.939545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23057e2-9e86-494d-b748-85602d25b124 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:04.967555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:04.975838 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76684cd-e91f-4393-86f1-40614760bb73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:04.976470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76684cd-e91f-4393-86f1-40614760bb73 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:04.976970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76684cd-e91f-4393-86f1-40614760bb73 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:04.977260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76684cd-e91f-4393-86f1-40614760bb73 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:04.984626 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76684cd-e91f-4393-86f1-40614760bb73 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:04.986069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76684cd-e91f-4393-86f1-40614760bb73 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:04.997262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-c76684cd-e91f-4393-86f1-40614760bb73 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 07 03:51:04.997262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:04.997262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:04.997262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:04.997262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:04.997262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:04.997262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:04.997262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:04.997262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:04.997262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:04.997262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:04.997262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:04.997262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:51:04.997262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:51:04.997262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:51:04.997262 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:04.998393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:04.999765 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:04.999765 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:04.999765 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:51:04.999765 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:51:04.999765 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:51:04.999765 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:51:04.999765 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:51:04.999765 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:51:04.999765 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 07 03:51:04.999765 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:51:04.999765 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 630/1253] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:51:04 2022] GET /identity/v3/users/placement => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:05.004161 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5b77ea-7914-49fa-806e-0614a0ed980d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:05.058518 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:05.102252 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5b77ea-7914-49fa-806e-0614a0ed980d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:05.137468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cd264e-3353-4524-b996-d11d0a0fff11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:05.138069 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cd264e-3353-4524-b996-d11d0a0fff11 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:05.138383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cd264e-3353-4524-b996-d11d0a0fff11 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:05.138659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cd264e-3353-4524-b996-d11d0a0fff11 None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:05.139386 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5cd264e-3353-4524-b996-d11d0a0fff11 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:05.139714 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5cd264e-3353-4524-b996-d11d0a0fff11 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:05.141111 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5cd264e-3353-4524-b996-d11d0a0fff11 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:05.157776 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 624/1254] 10.222.0.27 () {56 vars in 1143 bytes} [Wed Sep 7 03:51:05 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:05.163522 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76684cd-e91f-4393-86f1-40614760bb73 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:05.263025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76684cd-e91f-4393-86f1-40614760bb73 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:05.295348 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7f64ec-286f-4e36-993e-5b5a49587372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:05.295953 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7f64ec-286f-4e36-993e-5b5a49587372 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:05.296230 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7f64ec-286f-4e36-993e-5b5a49587372 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:05.296503 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7f64ec-286f-4e36-993e-5b5a49587372 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:05.350464 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:05.363147 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7f64ec-286f-4e36-993e-5b5a49587372 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:05.364588 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7f64ec-286f-4e36-993e-5b5a49587372 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:05.460084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 631/1255] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:51:05 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:51:05.466013 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cd264e-3353-4524-b996-d11d0a0fff11 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:05.563018 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cd264e-3353-4524-b996-d11d0a0fff11 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:05.578058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:05.603538 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d045efd8-7061-4d11-ba7e-3cee20d89f77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:05.604233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d045efd8-7061-4d11-ba7e-3cee20d89f77 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:05.604502 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d045efd8-7061-4d11-ba7e-3cee20d89f77 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:05.604852 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d045efd8-7061-4d11-ba7e-3cee20d89f77 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:05.616229 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d045efd8-7061-4d11-ba7e-3cee20d89f77 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:05.618539 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d045efd8-7061-4d11-ba7e-3cee20d89f77 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:05.633763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-d045efd8-7061-4d11-ba7e-3cee20d89f77 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:51:05.633763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:05.633763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:05.633763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:05.633763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:05.633763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:05.633763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:05.633763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:05.633763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:05.633763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:05.633763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:05.633763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:05.633763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:51:05.633763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:51:05.633763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:51:05.633763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:05.634951 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:51:05.636242 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:51:05.636242 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:51:05.636242 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:51:05.636242 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 625/1256] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:51:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:05.640580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7f64ec-286f-4e36-993e-5b5a49587372 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:05.724690 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:05.737410 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7f64ec-286f-4e36-993e-5b5a49587372 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:05.770522 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53639035-ccdb-4d94-8352-b1d9adbfbf01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:05.771118 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53639035-ccdb-4d94-8352-b1d9adbfbf01 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:05.771393 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53639035-ccdb-4d94-8352-b1d9adbfbf01 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:05.771663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53639035-ccdb-4d94-8352-b1d9adbfbf01 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:05.772293 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53639035-ccdb-4d94-8352-b1d9adbfbf01 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:05.772592 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53639035-ccdb-4d94-8352-b1d9adbfbf01 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:05.774261 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53639035-ccdb-4d94-8352-b1d9adbfbf01 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:05.785677 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 632/1257] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:51:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:05.791954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d045efd8-7061-4d11-ba7e-3cee20d89f77 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:05.889163 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d045efd8-7061-4d11-ba7e-3cee20d89f77 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:05.921625 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103f3a4f-0b6d-4ba5-ae72-c4ea63d8ab86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:05.922270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103f3a4f-0b6d-4ba5-ae72-c4ea63d8ab86 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:05.922556 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103f3a4f-0b6d-4ba5-ae72-c4ea63d8ab86 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:05.922836 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103f3a4f-0b6d-4ba5-ae72-c4ea63d8ab86 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:05.929272 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103f3a4f-0b6d-4ba5-ae72-c4ea63d8ab86 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:05.931254 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103f3a4f-0b6d-4ba5-ae72-c4ea63d8ab86 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:05.942267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-103f3a4f-0b6d-4ba5-ae72-c4ea63d8ab86 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:51:05.942267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:05.942267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:05.942267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:05.942267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:05.942267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:05.942267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:05.942267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:05.942267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:05.942267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:05.942267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:05.942267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:05.942267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:51:05.942267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:51:05.942267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:05.943276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:05.944623 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:51:05.944623 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:51:05.944623 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:51:05.944623 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:51:05.944623 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:51:05.944623 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:51:05.944623 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:51:05.944623 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:51:05.944623 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 626/1258] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:51:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:05.949691 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53639035-ccdb-4d94-8352-b1d9adbfbf01 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:06.046352 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53639035-ccdb-4d94-8352-b1d9adbfbf01 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:06.079133 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14479bc7-7721-4b01-b19a-689ffb7fa960 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:06.079727 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14479bc7-7721-4b01-b19a-689ffb7fa960 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:06.080006 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14479bc7-7721-4b01-b19a-689ffb7fa960 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:06.080542 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14479bc7-7721-4b01-b19a-689ffb7fa960 None admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:06.081219 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14479bc7-7721-4b01-b19a-689ffb7fa960 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:06.081527 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14479bc7-7721-4b01-b19a-689ffb7fa960 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:06.082968 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14479bc7-7721-4b01-b19a-689ffb7fa960 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:06.095534 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 633/1259] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:51:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:06.101464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103f3a4f-0b6d-4ba5-ae72-c4ea63d8ab86 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:06.127360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:06.200535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103f3a4f-0b6d-4ba5-ae72-c4ea63d8ab86 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:06.233561 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ddcb031-da30-423d-8a0e-4db5f935ba34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:06.234195 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ddcb031-da30-423d-8a0e-4db5f935ba34 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:06.234476 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ddcb031-da30-423d-8a0e-4db5f935ba34 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:06.234748 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ddcb031-da30-423d-8a0e-4db5f935ba34 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:06.235466 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ddcb031-da30-423d-8a0e-4db5f935ba34 None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=3bc3872e11a749758665f2c5dc1db8e9) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:06.235771 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ddcb031-da30-423d-8a0e-4db5f935ba34 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:06.237189 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ddcb031-da30-423d-8a0e-4db5f935ba34 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:06.243643 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 627/1260] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:51:06 2022] GET /identity/v3/role_assignments?user.id=3bc3872e11a749758665f2c5dc1db8e9&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:08.354730 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17373146-a669-4bf5-abbc-4a35dc5c7d3a None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:08.355283 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17373146-a669-4bf5-abbc-4a35dc5c7d3a None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:08.355283 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17373146-a669-4bf5-abbc-4a35dc5c7d3a None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:08.356627 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 634/1261] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:51:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:51:08.361673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0c28ff-5621-4cfa-b0a2-f67680963336 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:08.362089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0c28ff-5621-4cfa-b0a2-f67680963336 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:08.362237 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0c28ff-5621-4cfa-b0a2-f67680963336 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:08.379950 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-3a0c28ff-5621-4cfa-b0a2-f67680963336 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:51:08.381134 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-3a0c28ff-5621-4cfa-b0a2-f67680963336 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:51:08.444736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 628/1262] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:51:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 07 03:51:08.716533 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17373146-a669-4bf5-abbc-4a35dc5c7d3a None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:08.727405 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:08.816745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17373146-a669-4bf5-abbc-4a35dc5c7d3a None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:08.850549 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7da2d7-a0d3-487e-885d-658a8f6c8c8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:08.851176 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7da2d7-a0d3-487e-885d-658a8f6c8c8f None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:08.851453 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7da2d7-a0d3-487e-885d-658a8f6c8c8f None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:08.851792 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7da2d7-a0d3-487e-885d-658a8f6c8c8f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:08.861512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7da2d7-a0d3-487e-885d-658a8f6c8c8f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:51:08.866536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-9f7da2d7-a0d3-487e-885d-658a8f6c8c8f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:51:08.866536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:08.866536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:08.866536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:08.866536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:08.866536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:08.866536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:08.866536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:08.866536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:08.866536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:08.866536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:08.866536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:08.866536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:51:08.866536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:51:08.866536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:51:08.866536 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:51:08.868140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:51:08.868140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:51:08.868140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:51:08.868140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:51:08.868140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:08.868140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:08.868140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:08.868140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:08.868140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:08.868140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:08.868140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:08.868140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:08.868140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:08.868140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:08.868140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:08.868140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:08.869380 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:08.869380 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:08.869380 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:08.869380 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:08.869380 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:51:08.869380 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:51:08.869380 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:51:08.869380 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:51:08.869380 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:51:08.869380 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:51:08.869380 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:51:08.869380 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:51:08.869380 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 635/1263] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:51:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:08.874242 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0c28ff-5621-4cfa-b0a2-f67680963336 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:08.892461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:08.972499 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0c28ff-5621-4cfa-b0a2-f67680963336 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:09.005076 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c20d681-0a93-4a2d-8861-d82a183dde9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:09.005669 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c20d681-0a93-4a2d-8861-d82a183dde9b None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:09.005947 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c20d681-0a93-4a2d-8861-d82a183dde9b None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:09.006247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c20d681-0a93-4a2d-8861-d82a183dde9b None admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:09.006900 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c20d681-0a93-4a2d-8861-d82a183dde9b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:09.007199 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c20d681-0a93-4a2d-8861-d82a183dde9b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:09.009111 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c20d681-0a93-4a2d-8861-d82a183dde9b None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:09.016775 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 629/1264] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:51:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:51:09.022542 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7da2d7-a0d3-487e-885d-658a8f6c8c8f None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:09.088130 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:09.120591 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7da2d7-a0d3-487e-885d-658a8f6c8c8f None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:09.154941 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcce9abc-5366-443b-aded-e98dea894ef6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:09.155571 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcce9abc-5366-443b-aded-e98dea894ef6 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:09.155852 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcce9abc-5366-443b-aded-e98dea894ef6 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:09.156128 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcce9abc-5366-443b-aded-e98dea894ef6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:09.166106 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcce9abc-5366-443b-aded-e98dea894ef6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:09.167588 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcce9abc-5366-443b-aded-e98dea894ef6 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:09.182509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-bcce9abc-5366-443b-aded-e98dea894ef6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:51:09.182509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:09.182509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:09.182509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:09.182509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:09.182509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:09.182509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:09.182509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:09.182509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:09.182509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:09.182509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:09.182509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:09.182509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:51:09.182509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:51:09.182509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:51:09.182509 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:09.183667 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:51:09.184943 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:51:09.184943 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:51:09.184943 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:51:09.184943 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 636/1265] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:51:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:09.189415 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c20d681-0a93-4a2d-8861-d82a183dde9b None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:09.271561 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:09.291145 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c20d681-0a93-4a2d-8861-d82a183dde9b None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:09.323726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a53c273-bb69-4895-91ed-c599999860af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:09.324328 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a53c273-bb69-4895-91ed-c599999860af None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:09.324611 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a53c273-bb69-4895-91ed-c599999860af None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:09.324888 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a53c273-bb69-4895-91ed-c599999860af None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:09.325529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a53c273-bb69-4895-91ed-c599999860af None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:09.325835 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a53c273-bb69-4895-91ed-c599999860af None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:09.327229 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a53c273-bb69-4895-91ed-c599999860af None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:09.339669 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 630/1266] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:51:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:09.345350 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcce9abc-5366-443b-aded-e98dea894ef6 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:09.442554 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcce9abc-5366-443b-aded-e98dea894ef6 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:09.461792 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:09.475443 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a08e0d-f35c-4708-96dc-d6f17db611f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:09.476058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a08e0d-f35c-4708-96dc-d6f17db611f0 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:09.476337 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a08e0d-f35c-4708-96dc-d6f17db611f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:09.477209 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a08e0d-f35c-4708-96dc-d6f17db611f0 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:09.484559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a08e0d-f35c-4708-96dc-d6f17db611f0 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:09.485988 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a08e0d-f35c-4708-96dc-d6f17db611f0 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:09.497146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-45a08e0d-f35c-4708-96dc-d6f17db611f0 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 07 03:51:09.497146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:09.497146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:09.497146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:09.497146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:09.497146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:09.497146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:09.497146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:09.497146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:09.497146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:09.497146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:09.497146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:09.497146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:51:09.497146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:51:09.497146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:51:09.497146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:09.498254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:09.499183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:09.499183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:09.499183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:51:09.499183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:51:09.499183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:51:09.499183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:51:09.499183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:51:09.499183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:51:09.499183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 07 03:51:09.499183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:51:09.499183 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 637/1267] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:51:09 2022] GET /identity/v3/users/placement => generated 87 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:09.504140 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a53c273-bb69-4895-91ed-c599999860af None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:09.602556 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a53c273-bb69-4895-91ed-c599999860af None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:09.635977 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f7ab40-edc6-43a9-9018-cfb9b03c5e03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:09.636571 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f7ab40-edc6-43a9-9018-cfb9b03c5e03 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:09.636847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f7ab40-edc6-43a9-9018-cfb9b03c5e03 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:09.637117 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f7ab40-edc6-43a9-9018-cfb9b03c5e03 None admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:09.637840 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f7ab40-edc6-43a9-9018-cfb9b03c5e03 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:09.638165 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f7ab40-edc6-43a9-9018-cfb9b03c5e03 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:09.639577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f7ab40-edc6-43a9-9018-cfb9b03c5e03 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:09.650754 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:09.659553 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 631/1268] 10.222.0.27 () {56 vars in 1143 bytes} [Wed Sep 7 03:51:09 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:09.665438 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a08e0d-f35c-4708-96dc-d6f17db611f0 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:09.762774 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a08e0d-f35c-4708-96dc-d6f17db611f0 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:09.794512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e592ac-b14a-45d3-be04-974e97d52c1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:09.795125 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e592ac-b14a-45d3-be04-974e97d52c1b None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:09.795406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e592ac-b14a-45d3-be04-974e97d52c1b None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:09.795687 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e592ac-b14a-45d3-be04-974e97d52c1b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:09.806623 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e592ac-b14a-45d3-be04-974e97d52c1b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:09.808057 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e592ac-b14a-45d3-be04-974e97d52c1b None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:09.822360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-00e592ac-b14a-45d3-be04-974e97d52c1b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:51:09.822360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:09.822360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:09.822360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:09.822360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:09.822360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:09.822360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:09.822360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:09.822360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:09.822360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:09.822360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:09.822360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:09.822360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:51:09.822360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:51:09.822360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:51:09.822360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:09.823494 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:51:09.824782 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:51:09.824782 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:51:09.824782 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:51:09.824782 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 638/1269] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:51:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:09.829054 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f7ab40-edc6-43a9-9018-cfb9b03c5e03 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:09.925017 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f7ab40-edc6-43a9-9018-cfb9b03c5e03 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:09.958570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66d6093-6af4-4eea-9e25-866391c2b290 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:09.959165 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66d6093-6af4-4eea-9e25-866391c2b290 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:09.959447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66d6093-6af4-4eea-9e25-866391c2b290 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:09.959722 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66d6093-6af4-4eea-9e25-866391c2b290 None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:09.960360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66d6093-6af4-4eea-9e25-866391c2b290 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:09.960660 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66d6093-6af4-4eea-9e25-866391c2b290 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:09.962187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66d6093-6af4-4eea-9e25-866391c2b290 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:09.972995 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 632/1270] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:51:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:09.979383 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e592ac-b14a-45d3-be04-974e97d52c1b None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:10.011094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:10.077124 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e592ac-b14a-45d3-be04-974e97d52c1b None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:10.109580 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae248da-fcf7-40ff-bd03-4f2f0cc34129 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:10.110243 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae248da-fcf7-40ff-bd03-4f2f0cc34129 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:10.110520 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae248da-fcf7-40ff-bd03-4f2f0cc34129 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:10.110790 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae248da-fcf7-40ff-bd03-4f2f0cc34129 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:10.117083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae248da-fcf7-40ff-bd03-4f2f0cc34129 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:10.119099 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae248da-fcf7-40ff-bd03-4f2f0cc34129 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:10.130447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-7ae248da-fcf7-40ff-bd03-4f2f0cc34129 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:51:10.130447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:10.130447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:10.130447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:10.130447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:10.130447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:10.130447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:10.130447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:10.130447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:10.130447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:10.130447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:10.130447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:10.130447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:51:10.130447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:51:10.130447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:10.131470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:10.137384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:51:10.137384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:51:10.137384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:51:10.137384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:51:10.137384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:51:10.137384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:51:10.137384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:51:10.137384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:51:10.137384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 639/1271] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:51:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:10.138101 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66d6093-6af4-4eea-9e25-866391c2b290 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:10.194016 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:10.234805 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66d6093-6af4-4eea-9e25-866391c2b290 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:10.267708 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6be4e26-ac1f-45f7-bc8b-acb32a35e2d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:10.268311 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6be4e26-ac1f-45f7-bc8b-acb32a35e2d6 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:10.268591 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6be4e26-ac1f-45f7-bc8b-acb32a35e2d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:10.268867 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6be4e26-ac1f-45f7-bc8b-acb32a35e2d6 None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:10.269554 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6be4e26-ac1f-45f7-bc8b-acb32a35e2d6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:10.269857 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6be4e26-ac1f-45f7-bc8b-acb32a35e2d6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:10.271307 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6be4e26-ac1f-45f7-bc8b-acb32a35e2d6 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:10.282954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 633/1272] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:51:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:10.289566 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae248da-fcf7-40ff-bd03-4f2f0cc34129 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:10.378263 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:10.387416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae248da-fcf7-40ff-bd03-4f2f0cc34129 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:10.420590 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007dc3a1-ff13-4552-8f28-c7a684fb1b8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:10.421226 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007dc3a1-ff13-4552-8f28-c7a684fb1b8d None admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:10.421504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007dc3a1-ff13-4552-8f28-c7a684fb1b8d None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:10.421774 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007dc3a1-ff13-4552-8f28-c7a684fb1b8d None admin] PATH_INFO: `/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/3bc3872e11a749758665f2c5dc1db8e9/roles/9d5b19a61c46475d8048b1bd66ca25a1` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:10.423649 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007dc3a1-ff13-4552-8f28-c7a684fb1b8d None admin] RBAC: Authorizing `identity:create_grant(project_id=7c4816e95144485e8135e5b8f68c3662, user_id=3bc3872e11a749758665f2c5dc1db8e9, role_id=9d5b19a61c46475d8048b1bd66ca25a1)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:10.425066 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007dc3a1-ff13-4552-8f28-c7a684fb1b8d None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:10.438443 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 640/1273] 10.222.0.27 () {58 vars in 1439 bytes} [Wed Sep 7 03:51:10 2022] PUT /identity/v3/projects/7c4816e95144485e8135e5b8f68c3662/users/3bc3872e11a749758665f2c5dc1db8e9/roles/9d5b19a61c46475d8048b1bd66ca25a1 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:51:12.572420 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c222fe2-3759-40f4-aa88-a179cd6dc591 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:12.573033 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c222fe2-3759-40f4-aa88-a179cd6dc591 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:12.573033 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c222fe2-3759-40f4-aa88-a179cd6dc591 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:12.574191 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 634/1274] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:51:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:51:12.579915 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daabcfc-2152-4bc9-a903-d7e1dea001d6 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:12.580629 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daabcfc-2152-4bc9-a903-d7e1dea001d6 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:12.581026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daabcfc-2152-4bc9-a903-d7e1dea001d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:12.606414 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-8daabcfc-2152-4bc9-a903-d7e1dea001d6 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:51:12.607848 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-8daabcfc-2152-4bc9-a903-d7e1dea001d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:51:12.682525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 641/1275] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:51:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:51:12.949471 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c222fe2-3759-40f4-aa88-a179cd6dc591 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:13.047370 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c222fe2-3759-40f4-aa88-a179cd6dc591 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:13.064384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:13.081245 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c031bbd0-1793-493e-9863-d358da62d6ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:13.081862 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c031bbd0-1793-493e-9863-d358da62d6ec None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:13.082164 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c031bbd0-1793-493e-9863-d358da62d6ec None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:13.082442 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c031bbd0-1793-493e-9863-d358da62d6ec None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:13.092379 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c031bbd0-1793-493e-9863-d358da62d6ec None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 07 03:51:13.097271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-c031bbd0-1793-493e-9863-d358da62d6ec None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:51:13.097271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:13.097271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:13.097271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:13.097271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:13.097271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:13.097271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:13.097271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:13.097271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:13.097271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:13.097271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:13.097271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:13.097271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 07 03:51:13.097271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 07 03:51:13.097271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 07 03:51:13.097271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 07 03:51:13.098423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 07 03:51:13.098423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 07 03:51:13.098423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 07 03:51:13.098423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 07 03:51:13.098423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:13.098423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:13.098423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:13.098423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:13.098423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:13.098423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:13.098423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:13.098423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:13.098423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:13.098423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:13.098423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:13.098423 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:13.099496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:13.099496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:13.099496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:13.099496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:13.099496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 07 03:51:13.099496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 07 03:51:13.099496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 07 03:51:13.099496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 07 03:51:13.099496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 07 03:51:13.099496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 07 03:51:13.099496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 07 03:51:13.099496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:51:13.099496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 635/1276] 10.222.0.27 () {56 vars in 1096 bytes} [Wed Sep 7 03:51:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:13.104942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8daabcfc-2152-4bc9-a903-d7e1dea001d6 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:13.184991 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:13.203018 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8daabcfc-2152-4bc9-a903-d7e1dea001d6 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:13.236277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6143ed-ca40-41d9-b0d5-87d6da1f9737 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:13.236885 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6143ed-ca40-41d9-b0d5-87d6da1f9737 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:13.237167 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6143ed-ca40-41d9-b0d5-87d6da1f9737 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:13.237440 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6143ed-ca40-41d9-b0d5-87d6da1f9737 None admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:13.238099 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6143ed-ca40-41d9-b0d5-87d6da1f9737 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:13.238427 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6143ed-ca40-41d9-b0d5-87d6da1f9737 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:13.239788 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6143ed-ca40-41d9-b0d5-87d6da1f9737 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:13.248456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 642/1277] 10.222.0.27 () {56 vars in 1099 bytes} [Wed Sep 7 03:51:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:13.254212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c031bbd0-1793-493e-9863-d358da62d6ec None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:13.354241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c031bbd0-1793-493e-9863-d358da62d6ec None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:13.386698 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d076b476-fa28-4812-94ae-3604dff7d020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:13.387314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d076b476-fa28-4812-94ae-3604dff7d020 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:13.387593 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d076b476-fa28-4812-94ae-3604dff7d020 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:13.387863 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d076b476-fa28-4812-94ae-3604dff7d020 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:13.397372 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d076b476-fa28-4812-94ae-3604dff7d020 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:13.399520 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d076b476-fa28-4812-94ae-3604dff7d020 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:13.413511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-d076b476-fa28-4812-94ae-3604dff7d020 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:51:13.413511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:13.413511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:13.413511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:13.413511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:13.413511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:13.413511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:13.413511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:13.413511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:13.413511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:13.413511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:13.413511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:13.413511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:51:13.413511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:51:13.413511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:51:13.413511 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:13.415296 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:51:13.416617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:51:13.416617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:51:13.416617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:51:13.416617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 636/1278] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:51:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:13.421069 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6143ed-ca40-41d9-b0d5-87d6da1f9737 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:13.516414 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6143ed-ca40-41d9-b0d5-87d6da1f9737 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:13.548318 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e016dcce-718e-46cf-b206-c8d7aff96560 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:13.548905 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e016dcce-718e-46cf-b206-c8d7aff96560 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:13.549182 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e016dcce-718e-46cf-b206-c8d7aff96560 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:13.549451 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e016dcce-718e-46cf-b206-c8d7aff96560 None admin] PATH_INFO: `/v3/domains` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:13.550078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e016dcce-718e-46cf-b206-c8d7aff96560 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:13.550407 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e016dcce-718e-46cf-b206-c8d7aff96560 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:13.551751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e016dcce-718e-46cf-b206-c8d7aff96560 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:13.563668 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 643/1279] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:51:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:13.569212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d076b476-fa28-4812-94ae-3604dff7d020 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:13.601420 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:13.670127 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d076b476-fa28-4812-94ae-3604dff7d020 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:13.703033 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263b12e0-78c9-423a-a3a6-f30559478d8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:13.703644 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263b12e0-78c9-423a-a3a6-f30559478d8d None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:13.703920 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263b12e0-78c9-423a-a3a6-f30559478d8d None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:13.704189 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263b12e0-78c9-423a-a3a6-f30559478d8d None admin] PATH_INFO: `/v3/users/placement` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:13.711761 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263b12e0-78c9-423a-a3a6-f30559478d8d None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:13.713274 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263b12e0-78c9-423a-a3a6-f30559478d8d None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:13.724035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-263b12e0-78c9-423a-a3a6-f30559478d8d None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 07 03:51:13.724035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:13.724035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:13.724035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:13.724035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:13.724035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:13.724035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:13.724035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:13.724035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:13.724035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:13.724035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:13.724035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:13.724035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 07 03:51:13.724035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 07 03:51:13.724035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 07 03:51:13.724035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:13.725137 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:13.726175 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:13.726175 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:13.726175 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 07 03:51:13.726175 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 07 03:51:13.726175 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 07 03:51:13.726175 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 07 03:51:13.726175 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 07 03:51:13.726175 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 07 03:51:13.726175 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 07 03:51:13.726175 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:51:13.726175 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 637/1280] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:51:13 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:13.731542 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e016dcce-718e-46cf-b206-c8d7aff96560 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:13.735286 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:13.829441 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e016dcce-718e-46cf-b206-c8d7aff96560 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:13.862302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50c0f5d-3a82-441d-8544-58b180f9dbce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:13.862902 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50c0f5d-3a82-441d-8544-58b180f9dbce None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:13.863178 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50c0f5d-3a82-441d-8544-58b180f9dbce None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:13.863446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50c0f5d-3a82-441d-8544-58b180f9dbce None admin] PATH_INFO: `/v3/users` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:13.864182 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50c0f5d-3a82-441d-8544-58b180f9dbce None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:13.864483 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50c0f5d-3a82-441d-8544-58b180f9dbce None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:13.865902 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50c0f5d-3a82-441d-8544-58b180f9dbce None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:13.882923 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 644/1281] 10.222.0.27 () {56 vars in 1143 bytes} [Wed Sep 7 03:51:13 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:13.888668 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263b12e0-78c9-423a-a3a6-f30559478d8d None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:13.977064 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:13.987203 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263b12e0-78c9-423a-a3a6-f30559478d8d None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:14.019943 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2220ec-3e47-48cf-bacf-2a2189c29050 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:14.020537 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2220ec-3e47-48cf-bacf-2a2189c29050 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:14.020815 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2220ec-3e47-48cf-bacf-2a2189c29050 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:14.021089 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2220ec-3e47-48cf-bacf-2a2189c29050 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:14.087618 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2220ec-3e47-48cf-bacf-2a2189c29050 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:14.088975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2220ec-3e47-48cf-bacf-2a2189c29050 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:14.175676 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:14.186042 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 638/1282] 10.222.0.27 () {58 vars in 1282 bytes} [Wed Sep 7 03:51:13 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 07 03:51:14.191934 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50c0f5d-3a82-441d-8544-58b180f9dbce None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:14.252191 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:14.289932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50c0f5d-3a82-441d-8544-58b180f9dbce None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:14.321813 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bcab2f-ee92-486c-acc2-2b6347dea325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:14.322459 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bcab2f-ee92-486c-acc2-2b6347dea325 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:14.322732 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bcab2f-ee92-486c-acc2-2b6347dea325 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:14.323000 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bcab2f-ee92-486c-acc2-2b6347dea325 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:14.332767 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bcab2f-ee92-486c-acc2-2b6347dea325 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:14.334224 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bcab2f-ee92-486c-acc2-2b6347dea325 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:14.348750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-b3bcab2f-ee92-486c-acc2-2b6347dea325 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:51:14.348750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:14.348750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:14.348750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:14.348750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:14.348750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:14.348750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:14.348750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:14.348750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:14.348750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:14.348750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:14.348750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:14.348750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 07 03:51:14.348750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 07 03:51:14.348750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 07 03:51:14.348750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:14.349861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 07 03:51:14.350798 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 07 03:51:14.350798 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 07 03:51:14.350798 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:51:14.350798 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 645/1283] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:51:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:14.356235 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2220ec-3e47-48cf-bacf-2a2189c29050 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:14.451498 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2220ec-3e47-48cf-bacf-2a2189c29050 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:14.485416 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e4e0ce-c2bf-4657-b12d-1cb76cdba92f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:14.486005 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e4e0ce-c2bf-4657-b12d-1cb76cdba92f None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:14.486318 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e4e0ce-c2bf-4657-b12d-1cb76cdba92f None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:14.486594 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e4e0ce-c2bf-4657-b12d-1cb76cdba92f None admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:14.487231 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e4e0ce-c2bf-4657-b12d-1cb76cdba92f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:14.487535 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e4e0ce-c2bf-4657-b12d-1cb76cdba92f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:14.488886 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e4e0ce-c2bf-4657-b12d-1cb76cdba92f None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:14.499733 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 639/1284] 10.222.0.27 () {56 vars in 1109 bytes} [Wed Sep 7 03:51:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:14.505285 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bcab2f-ee92-486c-acc2-2b6347dea325 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:14.603256 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bcab2f-ee92-486c-acc2-2b6347dea325 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:14.616037 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:14.636332 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27e6632-def8-4fac-b4e8-5b019f6975f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:14.636950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27e6632-def8-4fac-b4e8-5b019f6975f4 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:14.637226 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27e6632-def8-4fac-b4e8-5b019f6975f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:14.637497 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27e6632-def8-4fac-b4e8-5b019f6975f4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:14.643797 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27e6632-def8-4fac-b4e8-5b019f6975f4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:14.645215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27e6632-def8-4fac-b4e8-5b019f6975f4 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:14.656925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application [None req-d27e6632-def8-4fac-b4e8-5b019f6975f4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:51:14.656925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:14.656925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:14.656925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:14.656925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:14.656925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:14.656925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:14.656925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:14.656925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:14.656925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:14.656925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:14.656925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:14.656925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 07 03:51:14.656925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 07 03:51:14.656925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:14.657942 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:14.659094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 07 03:51:14.659094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 07 03:51:14.659094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 07 03:51:14.659094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 07 03:51:14.659094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 07 03:51:14.659094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 07 03:51:14.659094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 07 03:51:14.659094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: ERROR keystone.server.flask.application Sep 07 03:51:14.659094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 646/1285] 10.222.0.27 () {56 vars in 1111 bytes} [Wed Sep 7 03:51:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:14.663788 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e4e0ce-c2bf-4657-b12d-1cb76cdba92f None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:14.712877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:14.760662 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e4e0ce-c2bf-4657-b12d-1cb76cdba92f None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:14.792927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c24fc64-170d-47da-9b11-5c0f2a123076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:14.793518 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c24fc64-170d-47da-9b11-5c0f2a123076 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:14.793797 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c24fc64-170d-47da-9b11-5c0f2a123076 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:14.794069 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c24fc64-170d-47da-9b11-5c0f2a123076 None admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:14.794787 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c24fc64-170d-47da-9b11-5c0f2a123076 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:14.795091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c24fc64-170d-47da-9b11-5c0f2a123076 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:14.796499 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c24fc64-170d-47da-9b11-5c0f2a123076 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:14.808062 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 640/1286] 10.222.0.27 () {56 vars in 1148 bytes} [Wed Sep 7 03:51:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:14.814011 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27e6632-def8-4fac-b4e8-5b019f6975f4 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:14.912110 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27e6632-def8-4fac-b4e8-5b019f6975f4 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:14.944424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a4c79b-bec6-4b3e-8645-f34a99d8fc43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:14.945022 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a4c79b-bec6-4b3e-8645-f34a99d8fc43 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:14.945856 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a4c79b-bec6-4b3e-8645-f34a99d8fc43 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:14.946142 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a4c79b-bec6-4b3e-8645-f34a99d8fc43 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:14.946860 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a4c79b-bec6-4b3e-8645-f34a99d8fc43 None admin] RBAC: Adding query filter params (role.id=9d5b19a61c46475d8048b1bd66ca25a1, scope.project.id=7c4816e95144485e8135e5b8f68c3662, user.id=3bc3872e11a749758665f2c5dc1db8e9) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:14.947168 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a4c79b-bec6-4b3e-8645-f34a99d8fc43 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:14.948589 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a4c79b-bec6-4b3e-8645-f34a99d8fc43 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:14.954189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 647/1287] 10.222.0.27 () {56 vars in 1374 bytes} [Wed Sep 7 03:51:14 2022] GET /identity/v3/role_assignments?user.id=3bc3872e11a749758665f2c5dc1db8e9&scope.project.id=7c4816e95144485e8135e5b8f68c3662&role.id=9d5b19a61c46475d8048b1bd66ca25a1 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:17.094645 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d97602a-1db5-4c2a-8ad2-24a827fce2ad None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:17.095178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d97602a-1db5-4c2a-8ad2-24a827fce2ad None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:17.095178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d97602a-1db5-4c2a-8ad2-24a827fce2ad None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:17.096513 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 641/1288] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:51:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:51:17.101693 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4ac643-aa84-4990-a364-54ba04051f4b None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:17.102194 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4ac643-aa84-4990-a364-54ba04051f4b None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:17.102486 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4ac643-aa84-4990-a364-54ba04051f4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:17.120969 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-ef4ac643-aa84-4990-a364-54ba04051f4b None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:51:17.122198 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-ef4ac643-aa84-4990-a364-54ba04051f4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:51:17.151572 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:17.196248 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 648/1289] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:51:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:51:17.469247 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d97602a-1db5-4c2a-8ad2-24a827fce2ad None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:17.568429 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d97602a-1db5-4c2a-8ad2-24a827fce2ad None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:17.579574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:17.605435 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cd3241-0a52-4c86-bde1-ec5458ceb353 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:17.606074 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cd3241-0a52-4c86-bde1-ec5458ceb353 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:17.606398 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cd3241-0a52-4c86-bde1-ec5458ceb353 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:17.606695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cd3241-0a52-4c86-bde1-ec5458ceb353 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:17.607306 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67cd3241-0a52-4c86-bde1-ec5458ceb353 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:17.608676 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67cd3241-0a52-4c86-bde1-ec5458ceb353 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-67cd3241-0a52-4c86-bde1-ec5458ceb353 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:17.617711 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:17.618884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:17.618884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:17.618884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:17.618884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:17.618884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:17.618884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:17.618884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:51:17.618884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:51:17.618884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:51:17.618884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:51:17.618884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:17.618884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:17.618884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:17.618884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:17.618884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:17.618884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:17.619745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:17.619745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:17.619745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:17.619745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:17.619745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:17.619745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:17.619745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:17.619745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:17.619745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:17.619745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:17.619745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:51:17.619745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:51:17.619745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 07 03:51:17.619745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:51:17.619745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 642/1290] 10.222.0.27 () {56 vars in 1117 bytes} [Wed Sep 7 03:51:17 2022] GET /identity/v3/services/placement => generated 90 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:17.626039 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4ac643-aa84-4990-a364-54ba04051f4b None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:17.726707 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4ac643-aa84-4990-a364-54ba04051f4b None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:17.759610 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6aea61-1ad1-4f3d-bd6f-5980cacd0831 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:17.760212 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6aea61-1ad1-4f3d-bd6f-5980cacd0831 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:17.760487 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6aea61-1ad1-4f3d-bd6f-5980cacd0831 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:17.760756 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6aea61-1ad1-4f3d-bd6f-5980cacd0831 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:17.761382 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6aea61-1ad1-4f3d-bd6f-5980cacd0831 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:17.761684 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6aea61-1ad1-4f3d-bd6f-5980cacd0831 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:17.763061 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6aea61-1ad1-4f3d-bd6f-5980cacd0831 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:17.767647 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 649/1291] 10.222.0.27 () {56 vars in 1116 bytes} [Wed Sep 7 03:51:17 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 07 03:51:17.773477 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cd3241-0a52-4c86-bde1-ec5458ceb353 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:17.871959 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cd3241-0a52-4c86-bde1-ec5458ceb353 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:17.904766 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86f7694-bc7c-4ef8-96d1-7b9960270467 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:17.905359 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86f7694-bc7c-4ef8-96d1-7b9960270467 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:17.905639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86f7694-bc7c-4ef8-96d1-7b9960270467 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:17.905911 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86f7694-bc7c-4ef8-96d1-7b9960270467 None admin] PATH_INFO: `/v3/services` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:17.906555 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86f7694-bc7c-4ef8-96d1-7b9960270467 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:17.906859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86f7694-bc7c-4ef8-96d1-7b9960270467 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:17.908201 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86f7694-bc7c-4ef8-96d1-7b9960270467 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:17.912471 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 643/1292] 10.222.0.27 () {56 vars in 1116 bytes} [Wed Sep 7 03:51:17 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:20.040112 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0336bcd-5b1b-492f-b1d3-86fa70cab451 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:20.040675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0336bcd-5b1b-492f-b1d3-86fa70cab451 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:20.040675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0336bcd-5b1b-492f-b1d3-86fa70cab451 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:20.041993 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 650/1293] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:51:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:51:20.047326 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487a1c65-3ab5-4759-b9e6-be33c4710222 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:20.047724 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487a1c65-3ab5-4759-b9e6-be33c4710222 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:20.047879 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487a1c65-3ab5-4759-b9e6-be33c4710222 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:20.066365 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-487a1c65-3ab5-4759-b9e6-be33c4710222 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:51:20.067561 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-487a1c65-3ab5-4759-b9e6-be33c4710222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:51:20.096872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:20.133693 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 644/1294] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:51:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:51:20.404089 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0336bcd-5b1b-492f-b1d3-86fa70cab451 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:20.434930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:20.504068 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0336bcd-5b1b-492f-b1d3-86fa70cab451 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:20.536925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a440b1f-cfdb-4982-87c4-ba65b6ca0c8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:20.537521 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a440b1f-cfdb-4982-87c4-ba65b6ca0c8c None admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:20.537799 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a440b1f-cfdb-4982-87c4-ba65b6ca0c8c None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:20.538070 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a440b1f-cfdb-4982-87c4-ba65b6ca0c8c None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:20.539355 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a440b1f-cfdb-4982-87c4-ba65b6ca0c8c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:20.540720 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a440b1f-cfdb-4982-87c4-ba65b6ca0c8c None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:20.554227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 651/1295] 10.222.0.27 () {60 vars in 1141 bytes} [Wed Sep 7 03:51:20 2022] POST /identity/v3/services => generated 246 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:51:22.732971 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82db8769-d3b3-4fde-a04a-cf80dd73a8fa None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:22.733529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82db8769-d3b3-4fde-a04a-cf80dd73a8fa None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:22.733529 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82db8769-d3b3-4fde-a04a-cf80dd73a8fa None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:22.734888 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 645/1296] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:51:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:51:22.739922 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a03f987-fb9b-4a73-bb54-4bb217345df0 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:22.740360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a03f987-fb9b-4a73-bb54-4bb217345df0 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:22.740456 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a03f987-fb9b-4a73-bb54-4bb217345df0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:22.759170 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-8a03f987-fb9b-4a73-bb54-4bb217345df0 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:51:22.760347 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-8a03f987-fb9b-4a73-bb54-4bb217345df0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:51:22.830896 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 652/1297] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:51:22 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:51:23.098920 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82db8769-d3b3-4fde-a04a-cf80dd73a8fa None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:23.198678 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82db8769-d3b3-4fde-a04a-cf80dd73a8fa None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:23.232004 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937d4567-ecf7-47e1-ad03-7af2649bd157 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:23.232617 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937d4567-ecf7-47e1-ad03-7af2649bd157 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:23.232894 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937d4567-ecf7-47e1-ad03-7af2649bd157 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:23.233165 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937d4567-ecf7-47e1-ad03-7af2649bd157 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:23.233744 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937d4567-ecf7-47e1-ad03-7af2649bd157 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:23.235151 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937d4567-ecf7-47e1-ad03-7af2649bd157 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-937d4567-ecf7-47e1-ad03-7af2649bd157 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:23.244690 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:23.245875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:23.245875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:23.245875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:23.245875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:23.245875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:23.245875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:23.245875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:51:23.245875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:51:23.245875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:51:23.245875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:51:23.245875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:23.245875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:23.245875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:23.245875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:23.245875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:23.245875 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:23.246937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:23.246937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:23.246937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:23.246937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:23.246937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:23.246937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:23.246937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:23.246937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:23.246937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:23.246937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:23.246937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:51:23.246937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:51:23.246937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 07 03:51:23.246937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:51:23.246937 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 646/1298] 10.222.0.27 () {56 vars in 1117 bytes} [Wed Sep 7 03:51:23 2022] GET /identity/v3/services/placement => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:23.251799 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a03f987-fb9b-4a73-bb54-4bb217345df0 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:23.270339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:23.349462 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a03f987-fb9b-4a73-bb54-4bb217345df0 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:23.382316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1c5659-362c-4453-8b77-bd5f2bc7cf90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:23.383730 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1c5659-362c-4453-8b77-bd5f2bc7cf90 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:23.384011 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1c5659-362c-4453-8b77-bd5f2bc7cf90 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:23.384285 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1c5659-362c-4453-8b77-bd5f2bc7cf90 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:23.384908 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1c5659-362c-4453-8b77-bd5f2bc7cf90 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:23.385210 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1c5659-362c-4453-8b77-bd5f2bc7cf90 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:23.386600 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1c5659-362c-4453-8b77-bd5f2bc7cf90 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:23.391548 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 653/1299] 10.222.0.27 () {56 vars in 1116 bytes} [Wed Sep 7 03:51:23 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:23.397528 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937d4567-ecf7-47e1-ad03-7af2649bd157 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:23.428991 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:23.496886 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937d4567-ecf7-47e1-ad03-7af2649bd157 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:23.528767 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80d9bd1-7639-4e09-a5c0-51329c2159c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:23.529351 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80d9bd1-7639-4e09-a5c0-51329c2159c2 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:23.529628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80d9bd1-7639-4e09-a5c0-51329c2159c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:23.529896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80d9bd1-7639-4e09-a5c0-51329c2159c2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:23.530613 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80d9bd1-7639-4e09-a5c0-51329c2159c2 None admin] RBAC: Adding query filter params (interface=public, service_id=06d6183c75e842f5bc9e1e527c394582, region_id=RegionOne) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:23.530918 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80d9bd1-7639-4e09-a5c0-51329c2159c2 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:23.532251 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80d9bd1-7639-4e09-a5c0-51329c2159c2 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:23.536998 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 647/1300] 10.222.0.27 () {56 vars in 1251 bytes} [Wed Sep 7 03:51:23 2022] GET /identity/v3/endpoints?service_id=06d6183c75e842f5bc9e1e527c394582&interface=public®ion_id=RegionOne => generated 194 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:23.543259 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1c5659-362c-4453-8b77-bd5f2bc7cf90 None admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:23.639544 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:23.658042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1c5659-362c-4453-8b77-bd5f2bc7cf90 None admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:23.691799 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb49545c-0971-432f-aa18-b390f4cd0ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:23.692400 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb49545c-0971-432f-aa18-b390f4cd0ded None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:23.692694 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb49545c-0971-432f-aa18-b390f4cd0ded None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:23.692947 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb49545c-0971-432f-aa18-b390f4cd0ded None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:23.693538 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb49545c-0971-432f-aa18-b390f4cd0ded None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:23.695002 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb49545c-0971-432f-aa18-b390f4cd0ded None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:23.701826 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 654/1301] 10.222.0.27 () {56 vars in 1087 bytes} [Wed Sep 7 03:51:23 2022] GET /identity/v3/services => generated 2191 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 07 03:51:25.826610 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e141f1-5123-4089-9888-f4454e89b8e0 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:25.827169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e141f1-5123-4089-9888-f4454e89b8e0 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:25.827169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e141f1-5123-4089-9888-f4454e89b8e0 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:25.828488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 648/1302] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:51:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:51:25.833837 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a301c018-e11f-4580-996d-143fa1788884 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:25.834275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a301c018-e11f-4580-996d-143fa1788884 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:25.834414 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a301c018-e11f-4580-996d-143fa1788884 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:25.853003 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-a301c018-e11f-4580-996d-143fa1788884 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:51:25.854235 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-a301c018-e11f-4580-996d-143fa1788884 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:51:25.920231 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 655/1303] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:51:25 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 07 03:51:26.190758 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e141f1-5123-4089-9888-f4454e89b8e0 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:26.286852 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:26.293876 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e141f1-5123-4089-9888-f4454e89b8e0 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:26.329599 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299749e6-6f6b-418e-84ff-eed10a197fa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:26.330249 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299749e6-6f6b-418e-84ff-eed10a197fa1 None admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:26.330786 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299749e6-6f6b-418e-84ff-eed10a197fa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:26.331059 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299749e6-6f6b-418e-84ff-eed10a197fa1 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:26.331637 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299749e6-6f6b-418e-84ff-eed10a197fa1 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:26.332982 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299749e6-6f6b-418e-84ff-eed10a197fa1 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application [None req-299749e6-6f6b-418e-84ff-eed10a197fa1 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 07 03:51:26.341828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 07 03:51:26.343026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 07 03:51:26.343026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 07 03:51:26.343026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 07 03:51:26.343026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 07 03:51:26.343026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 07 03:51:26.343026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 07 03:51:26.343026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 07 03:51:26.343026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 07 03:51:26.343026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 07 03:51:26.343026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 07 03:51:26.343026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 07 03:51:26.343026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 07 03:51:26.343026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 07 03:51:26.343026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 07 03:51:26.343026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 07 03:51:26.343026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.get_or_create( Sep 07 03:51:26.344043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 07 03:51:26.344043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application with Lock( Sep 07 03:51:26.344043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 07 03:51:26.344043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self._enter() Sep 07 03:51:26.344043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 07 03:51:26.344043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 07 03:51:26.344043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 07 03:51:26.344043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application return self.creator() Sep 07 03:51:26.344043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 07 03:51:26.344043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application created_value = creator( Sep 07 03:51:26.344043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 07 03:51:26.344043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 07 03:51:26.344043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 07 03:51:26.344043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: ERROR keystone.server.flask.application Sep 07 03:51:26.344043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 649/1304] 10.222.0.27 () {56 vars in 1117 bytes} [Wed Sep 7 03:51:26 2022] GET /identity/v3/services/placement => generated 90 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 07 03:51:26.348921 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a301c018-e11f-4580-996d-143fa1788884 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:26.421729 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:26.444601 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a301c018-e11f-4580-996d-143fa1788884 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:26.478410 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3903b2-7b8d-4f6f-a78b-64ca22135789 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:26.479002 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3903b2-7b8d-4f6f-a78b-64ca22135789 None admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:26.479276 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3903b2-7b8d-4f6f-a78b-64ca22135789 None admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:26.479545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3903b2-7b8d-4f6f-a78b-64ca22135789 None admin] PATH_INFO: `/v3/services` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:26.480154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3903b2-7b8d-4f6f-a78b-64ca22135789 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81391) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:51:26.480451 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3903b2-7b8d-4f6f-a78b-64ca22135789 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:26.481801 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3903b2-7b8d-4f6f-a78b-64ca22135789 None admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:26.486725 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 656/1305] 10.222.0.27 () {56 vars in 1116 bytes} [Wed Sep 7 03:51:26 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:51:26.492468 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299749e6-6f6b-418e-84ff-eed10a197fa1 None admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:51:26.589570 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299749e6-6f6b-418e-84ff-eed10a197fa1 None admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:51:26.621726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e37d27-1b3c-470f-92e4-cb2d9b526908 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:51:26.622364 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e37d27-1b3c-470f-92e4-cb2d9b526908 None admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:51:26.622650 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e37d27-1b3c-470f-92e4-cb2d9b526908 None admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:51:26.622927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e37d27-1b3c-470f-92e4-cb2d9b526908 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:51:26.623586 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e37d27-1b3c-470f-92e4-cb2d9b526908 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:51:26.624942 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e37d27-1b3c-470f-92e4-cb2d9b526908 None admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:51:26.644153 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 650/1306] 10.222.0.27 () {60 vars in 1144 bytes} [Wed Sep 7 03:51:26 2022] POST /identity/v3/endpoints => generated 331 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:51:36.483858 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=30 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:51:36.640889 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:30.690550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ef173d-fbf4-4601-9a78-2840b9120591 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:30.691583 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ef173d-fbf4-4601-9a78-2840b9120591 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:30.692163 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ef173d-fbf4-4601-9a78-2840b9120591 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:30.700273 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8731b210-47b2-4690-9601-beb75455ae0d None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:30.700699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8731b210-47b2-4690-9601-beb75455ae0d None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:30.700822 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8731b210-47b2-4690-9601-beb75455ae0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:30.704451 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 657/1307] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:52:30 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:52:30.828326 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-8731b210-47b2-4690-9601-beb75455ae0d None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:52:30.829877 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-8731b210-47b2-4690-9601-beb75455ae0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:30.958195 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 651/1308] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:52:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 07 03:52:31.263315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de59ee08-d989-4ddd-8e3d-d65dc1a1e522 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:31.263684 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de59ee08-d989-4ddd-8e3d-d65dc1a1e522 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:31.264031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de59ee08-d989-4ddd-8e3d-d65dc1a1e522 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:31.265487 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 658/1309] 10.222.0.27 () {54 vars in 965 bytes} [Wed Sep 7 03:52:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:52:31.272477 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ff0cad-691f-491a-808e-099f8c437a86 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:31.272905 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ff0cad-691f-491a-808e-099f8c437a86 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:31.273319 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ff0cad-691f-491a-808e-099f8c437a86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:31.360581 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-f6ff0cad-691f-491a-808e-099f8c437a86 None None] MFA Rules not processed for user `77aac66a6ab44fc1a80ff2fd1572254b`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:52:31.361793 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-f6ff0cad-691f-491a-808e-099f8c437a86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:31.386926 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:31.446913 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 652/1310] 10.222.0.27 () {58 vars in 1062 bytes} [Wed Sep 7 03:52:31 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:52:31.455977 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de59ee08-d989-4ddd-8e3d-d65dc1a1e522 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:52:31.458060 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-de59ee08-d989-4ddd-8e3d-d65dc1a1e522 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:31.573322 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de59ee08-d989-4ddd-8e3d-d65dc1a1e522 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:52:31.654547 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c30538-0381-4bf3-a72d-1d7b42f1c93e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:52:31.655084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c30538-0381-4bf3-a72d-1d7b42f1c93e service neutron] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:31.655365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c30538-0381-4bf3-a72d-1d7b42f1c93e service neutron] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:31.655646 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c30538-0381-4bf3-a72d-1d7b42f1c93e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:31.657254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-b6c30538-0381-4bf3-a72d-1d7b42f1c93e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:31.714774 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:31.774712 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c30538-0381-4bf3-a72d-1d7b42f1c93e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:52:31.776244 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c30538-0381-4bf3-a72d-1d7b42f1c93e service neutron] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:52:31.795584 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 659/1311] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:52:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:52:34.267070 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3a080e-8cb7-4c88-9b40-655133034664 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:34.267564 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3a080e-8cb7-4c88-9b40-655133034664 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:34.267658 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3a080e-8cb7-4c88-9b40-655133034664 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:34.268974 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 653/1312] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:52:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:52:34.274653 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08a2084-290a-4cd8-9a9d-61705f69bb05 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:34.275094 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08a2084-290a-4cd8-9a9d-61705f69bb05 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:34.275204 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08a2084-290a-4cd8-9a9d-61705f69bb05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:34.347439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-d08a2084-290a-4cd8-9a9d-61705f69bb05 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:52:34.348656 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-d08a2084-290a-4cd8-9a9d-61705f69bb05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:34.450847 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 660/1313] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:52:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:52:34.733487 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3a080e-8cb7-4c88-9b40-655133034664 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:52:34.735051 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-7d3a080e-8cb7-4c88-9b40-655133034664 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:34.850606 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3a080e-8cb7-4c88-9b40-655133034664 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:52:34.927040 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:34.929872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc164e3-5050-4c8b-9167-a5555ff8f44a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:52:34.930543 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc164e3-5050-4c8b-9167-a5555ff8f44a service neutron] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:34.931090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc164e3-5050-4c8b-9167-a5555ff8f44a service neutron] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:34.931613 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc164e3-5050-4c8b-9167-a5555ff8f44a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:34.933460 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-9dc164e3-5050-4c8b-9167-a5555ff8f44a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:35.047946 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc164e3-5050-4c8b-9167-a5555ff8f44a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:52:35.049481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc164e3-5050-4c8b-9167-a5555ff8f44a service neutron] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:52:35.068794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 654/1314] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:52:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:52:37.591672 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c106db70-065d-4d18-8830-d57a2ef1ba32 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:37.592237 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c106db70-065d-4d18-8830-d57a2ef1ba32 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:37.592237 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c106db70-065d-4d18-8830-d57a2ef1ba32 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:37.593567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 661/1315] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:52:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:52:37.599034 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34eaa7c-88b4-45a0-bb44-c66dc8c755e5 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:37.600212 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34eaa7c-88b4-45a0-bb44-c66dc8c755e5 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:37.600495 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34eaa7c-88b4-45a0-bb44-c66dc8c755e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:37.670600 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-b34eaa7c-88b4-45a0-bb44-c66dc8c755e5 None None] MFA Rules not processed for user `65508fe717ca4eb996135023e2296dd1`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:52:37.672751 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-b34eaa7c-88b4-45a0-bb44-c66dc8c755e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:37.728797 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:37.773565 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 655/1316] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:52:37 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:52:38.057821 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c106db70-065d-4d18-8830-d57a2ef1ba32 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:52:38.059401 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-c106db70-065d-4d18-8830-d57a2ef1ba32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:38.069310 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:38.171033 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c106db70-065d-4d18-8830-d57a2ef1ba32 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:52:38.247659 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77ca9d0-efc7-4c4d-85aa-6d8b80ad5548 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:52:38.248271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77ca9d0-efc7-4c4d-85aa-6d8b80ad5548 service neutron] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:38.248551 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77ca9d0-efc7-4c4d-85aa-6d8b80ad5548 service neutron] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:38.248829 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77ca9d0-efc7-4c4d-85aa-6d8b80ad5548 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:38.250481 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-d77ca9d0-efc7-4c4d-85aa-6d8b80ad5548 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:38.311454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:38.357343 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77ca9d0-efc7-4c4d-85aa-6d8b80ad5548 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:52:38.358817 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77ca9d0-efc7-4c4d-85aa-6d8b80ad5548 service neutron] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:52:38.376848 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 662/1317] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:52:38 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 07 03:52:41.108663 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82ec283-067f-4210-98ea-969ecc1084cc None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:41.109251 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82ec283-067f-4210-98ea-969ecc1084cc None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:41.109251 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82ec283-067f-4210-98ea-969ecc1084cc None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:41.110597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 656/1318] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:52:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:52:41.116463 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f558ee4-0e9f-4661-b3b5-86eab0605113 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:41.116866 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f558ee4-0e9f-4661-b3b5-86eab0605113 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:41.116986 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f558ee4-0e9f-4661-b3b5-86eab0605113 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:41.181355 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-8f558ee4-0e9f-4661-b3b5-86eab0605113 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:52:41.182603 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-8f558ee4-0e9f-4661-b3b5-86eab0605113 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:41.279216 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 663/1319] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:52:41 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:52:41.562253 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82ec283-067f-4210-98ea-969ecc1084cc None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:52:41.563789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-e82ec283-067f-4210-98ea-969ecc1084cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:41.666661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82ec283-067f-4210-98ea-969ecc1084cc None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:52:41.738315 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5e814a-1901-4803-a217-b67241fdcdb5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:52:41.738924 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5e814a-1901-4803-a217-b67241fdcdb5 service neutron] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:41.739205 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5e814a-1901-4803-a217-b67241fdcdb5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:41.739481 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5e814a-1901-4803-a217-b67241fdcdb5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:41.741099 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-1b5e814a-1901-4803-a217-b67241fdcdb5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:41.748434 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:41.848962 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5e814a-1901-4803-a217-b67241fdcdb5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:52:41.850577 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5e814a-1901-4803-a217-b67241fdcdb5 service neutron] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:52:41.868944 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 657/1320] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:52:41 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:52:44.672943 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8623c092-9d63-449a-b81d-5d7b6049113d None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:44.673515 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8623c092-9d63-449a-b81d-5d7b6049113d None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:44.673515 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8623c092-9d63-449a-b81d-5d7b6049113d None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:44.674895 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 664/1321] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:52:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:52:44.680562 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4e9d1-5229-4f7d-a093-a4672509a146 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:44.680957 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4e9d1-5229-4f7d-a093-a4672509a146 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:44.681233 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4e9d1-5229-4f7d-a093-a4672509a146 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:44.758534 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-a8f4e9d1-5229-4f7d-a093-a4672509a146 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:52:44.759799 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-a8f4e9d1-5229-4f7d-a093-a4672509a146 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:44.821526 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:44.863697 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 658/1322] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:52:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:52:45.170189 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435f9aae-fe33-454f-ada5-f9c639ad4ab4 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:45.170469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435f9aae-fe33-454f-ada5-f9c639ad4ab4 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:45.170780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435f9aae-fe33-454f-ada5-f9c639ad4ab4 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:45.172125 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 665/1323] 10.222.0.27 () {54 vars in 965 bytes} [Wed Sep 7 03:52:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:52:45.178464 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f635e898-ebdf-494e-bed2-568b11fe9502 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:45.178750 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f635e898-ebdf-494e-bed2-568b11fe9502 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:45.179023 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f635e898-ebdf-494e-bed2-568b11fe9502 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:45.257842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-f635e898-ebdf-494e-bed2-568b11fe9502 None None] MFA Rules not processed for user `77aac66a6ab44fc1a80ff2fd1572254b`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:52:45.259035 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-f635e898-ebdf-494e-bed2-568b11fe9502 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:45.341510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 659/1324] 10.222.0.27 () {58 vars in 1062 bytes} [Wed Sep 7 03:52:45 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 07 03:52:45.351177 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435f9aae-fe33-454f-ada5-f9c639ad4ab4 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:52:45.352874 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-435f9aae-fe33-454f-ada5-f9c639ad4ab4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:45.373449 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:45.456132 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435f9aae-fe33-454f-ada5-f9c639ad4ab4 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:52:45.531741 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9fb174-582c-4217-9390-05974e836d25 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:52:45.532343 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9fb174-582c-4217-9390-05974e836d25 service neutron] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:45.532624 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9fb174-582c-4217-9390-05974e836d25 service neutron] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:45.532905 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9fb174-582c-4217-9390-05974e836d25 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:45.534517 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-9b9fb174-582c-4217-9390-05974e836d25 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:45.606330 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:45.646400 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9fb174-582c-4217-9390-05974e836d25 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:52:45.647860 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9fb174-582c-4217-9390-05974e836d25 service neutron] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:52:45.666346 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 666/1325] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:52:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:52:48.126058 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5905ae9-4c3d-41cb-a62f-a19937193f5c None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:48.126618 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5905ae9-4c3d-41cb-a62f-a19937193f5c None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:48.126618 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5905ae9-4c3d-41cb-a62f-a19937193f5c None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:48.127900 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 660/1326] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:52:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:52:48.133143 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7572d440-371b-4975-878f-813173b5632c None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:48.133583 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7572d440-371b-4975-878f-813173b5632c None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:48.133708 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7572d440-371b-4975-878f-813173b5632c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:48.202412 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-7572d440-371b-4975-878f-813173b5632c None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:52:48.203614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-7572d440-371b-4975-878f-813173b5632c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:48.302173 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 667/1327] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:52:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:52:48.589149 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5905ae9-4c3d-41cb-a62f-a19937193f5c None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:52:48.590842 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-f5905ae9-4c3d-41cb-a62f-a19937193f5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:48.652049 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:48.696060 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5905ae9-4c3d-41cb-a62f-a19937193f5c None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:52:48.769380 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0043221-47e4-420c-8a55-1bf6d2544e9c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:52:48.769987 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0043221-47e4-420c-8a55-1bf6d2544e9c service neutron] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:48.770312 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0043221-47e4-420c-8a55-1bf6d2544e9c service neutron] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:48.770592 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0043221-47e4-420c-8a55-1bf6d2544e9c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:48.772283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-f0043221-47e4-420c-8a55-1bf6d2544e9c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:48.880954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:48.883314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0043221-47e4-420c-8a55-1bf6d2544e9c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:52:48.884795 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0043221-47e4-420c-8a55-1bf6d2544e9c service neutron] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:52:48.907726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 661/1328] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:52:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:52:51.155170 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31488133-5007-420f-bf95-2ea618435d1f None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:51.155670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31488133-5007-420f-bf95-2ea618435d1f None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:51.155787 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31488133-5007-420f-bf95-2ea618435d1f None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:51.157238 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 668/1329] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:52:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:52:51.162850 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64d1f5b-6f70-4222-92fd-0ef50d623582 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:51.163261 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64d1f5b-6f70-4222-92fd-0ef50d623582 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:51.163407 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64d1f5b-6f70-4222-92fd-0ef50d623582 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:51.231810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-d64d1f5b-6f70-4222-92fd-0ef50d623582 None None] MFA Rules not processed for user `65508fe717ca4eb996135023e2296dd1`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:52:51.233010 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-d64d1f5b-6f70-4222-92fd-0ef50d623582 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:51.332146 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 662/1330] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:52:51 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:52:51.621349 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31488133-5007-420f-bf95-2ea618435d1f None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:52:51.622723 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-31488133-5007-420f-bf95-2ea618435d1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:51.656414 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:51.731525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31488133-5007-420f-bf95-2ea618435d1f None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:52:51.806970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16df8a89-6dc5-44bc-8d18-7b9cbb315067 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:52:51.807593 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16df8a89-6dc5-44bc-8d18-7b9cbb315067 service neutron] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:51.807872 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16df8a89-6dc5-44bc-8d18-7b9cbb315067 service neutron] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:51.808147 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16df8a89-6dc5-44bc-8d18-7b9cbb315067 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:51.809757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-16df8a89-6dc5-44bc-8d18-7b9cbb315067 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:51.889936 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:51.927335 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16df8a89-6dc5-44bc-8d18-7b9cbb315067 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:52:51.928827 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16df8a89-6dc5-44bc-8d18-7b9cbb315067 service neutron] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:52:51.948793 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 669/1331] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:52:51 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:52:54.490313 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c353b0f2-8dea-44f7-8859-c48052ef2af5 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:54.490854 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c353b0f2-8dea-44f7-8859-c48052ef2af5 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:54.490969 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c353b0f2-8dea-44f7-8859-c48052ef2af5 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:54.492250 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 663/1332] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:52:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:52:54.497493 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b900f-4023-4297-bf3c-8c65836861ef None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:54.497905 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b900f-4023-4297-bf3c-8c65836861ef None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:54.498049 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b900f-4023-4297-bf3c-8c65836861ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:54.563950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-891b900f-4023-4297-bf3c-8c65836861ef None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:52:54.565103 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-891b900f-4023-4297-bf3c-8c65836861ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:54.661829 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 670/1333] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:52:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:52:54.942196 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c353b0f2-8dea-44f7-8859-c48052ef2af5 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:52:54.944298 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-c353b0f2-8dea-44f7-8859-c48052ef2af5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:55.013767 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:55.047095 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c353b0f2-8dea-44f7-8859-c48052ef2af5 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:52:55.117521 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4052c662-934b-48da-8dca-bfd27269f818 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:52:55.118144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4052c662-934b-48da-8dca-bfd27269f818 service neutron] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:55.118420 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4052c662-934b-48da-8dca-bfd27269f818 service neutron] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:55.118687 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4052c662-934b-48da-8dca-bfd27269f818 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:55.120258 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-4052c662-934b-48da-8dca-bfd27269f818 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:55.225643 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4052c662-934b-48da-8dca-bfd27269f818 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:52:55.227006 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4052c662-934b-48da-8dca-bfd27269f818 service neutron] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:52:55.236685 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:55.246049 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 664/1334] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:52:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:52:57.432539 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98de19d-b226-44ee-8d3c-be83f7d0b50e None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:57.433071 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98de19d-b226-44ee-8d3c-be83f7d0b50e None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:57.433071 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98de19d-b226-44ee-8d3c-be83f7d0b50e None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:57.434482 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 671/1335] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:52:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:52:57.439747 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9827f6ea-6c5e-44dd-bb7f-f9c7b4d51384 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:57.440144 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9827f6ea-6c5e-44dd-bb7f-f9c7b4d51384 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:57.440300 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9827f6ea-6c5e-44dd-bb7f-f9c7b4d51384 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:57.507356 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-9827f6ea-6c5e-44dd-bb7f-f9c7b4d51384 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:52:57.508566 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-9827f6ea-6c5e-44dd-bb7f-f9c7b4d51384 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:57.604750 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 665/1336] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:52:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:52:57.887993 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98de19d-b226-44ee-8d3c-be83f7d0b50e None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:52:57.889532 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-b98de19d-b226-44ee-8d3c-be83f7d0b50e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:57.928819 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:57.992293 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98de19d-b226-44ee-8d3c-be83f7d0b50e None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:52:58.065788 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e858cca-80f7-491a-9e6d-bbc252edfdc0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:52:58.066426 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e858cca-80f7-491a-9e6d-bbc252edfdc0 service neutron] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:52:58.066708 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e858cca-80f7-491a-9e6d-bbc252edfdc0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:52:58.067006 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e858cca-80f7-491a-9e6d-bbc252edfdc0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:52:58.068979 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-9e858cca-80f7-491a-9e6d-bbc252edfdc0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:52:58.150971 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:52:58.177548 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e858cca-80f7-491a-9e6d-bbc252edfdc0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:52:58.179289 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e858cca-80f7-491a-9e6d-bbc252edfdc0 service neutron] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:52:58.196899 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 672/1337] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:52:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:53:00.634040 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6f5f0c-bdd5-45ec-916a-a4761ce170af None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:00.634634 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6f5f0c-bdd5-45ec-916a-a4761ce170af None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:00.634634 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6f5f0c-bdd5-45ec-916a-a4761ce170af None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:00.635946 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 666/1338] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:53:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:00.641002 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77091f7-2fc6-484f-888f-8905fd0d7c53 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:00.641392 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77091f7-2fc6-484f-888f-8905fd0d7c53 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:00.641571 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77091f7-2fc6-484f-888f-8905fd0d7c53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:00.706970 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-e77091f7-2fc6-484f-888f-8905fd0d7c53 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:00.708780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-e77091f7-2fc6-484f-888f-8905fd0d7c53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:00.807584 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 673/1339] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:53:00 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:01.097671 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6f5f0c-bdd5-45ec-916a-a4761ce170af None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:53:01.100260 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-7b6f5f0c-bdd5-45ec-916a-a4761ce170af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:01.179019 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:01.207425 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6f5f0c-bdd5-45ec-916a-a4761ce170af None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:53:01.280568 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7008617-ea34-4c90-8eb4-be4e8b2c400e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:53:01.281174 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7008617-ea34-4c90-8eb4-be4e8b2c400e service neutron] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:01.281456 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7008617-ea34-4c90-8eb4-be4e8b2c400e service neutron] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:01.281733 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7008617-ea34-4c90-8eb4-be4e8b2c400e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:01.283366 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-d7008617-ea34-4c90-8eb4-be4e8b2c400e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:01.391833 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7008617-ea34-4c90-8eb4-be4e8b2c400e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:53:01.393177 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7008617-ea34-4c90-8eb4-be4e8b2c400e service neutron] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:53:01.410971 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 667/1340] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:53:01 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:53:05.524856 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2a4fd3-eecb-45b9-ae77-e9375ef2ebec None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:05.525384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2a4fd3-eecb-45b9-ae77-e9375ef2ebec None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:05.525384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2a4fd3-eecb-45b9-ae77-e9375ef2ebec None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:05.526698 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 674/1341] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:53:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:05.531938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7361253-e41d-461d-a78e-393d9191d098 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:05.532384 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7361253-e41d-461d-a78e-393d9191d098 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:05.532490 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7361253-e41d-461d-a78e-393d9191d098 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:05.538703 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:05.602771 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-b7361253-e41d-461d-a78e-393d9191d098 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:05.603920 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-b7361253-e41d-461d-a78e-393d9191d098 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:05.704203 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 668/1342] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:53:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:05.990968 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2a4fd3-eecb-45b9-ae77-e9375ef2ebec None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:53:05.992496 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-0a2a4fd3-eecb-45b9-ae77-e9375ef2ebec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:06.041145 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:06.097351 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2a4fd3-eecb-45b9-ae77-e9375ef2ebec None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:53:06.170478 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c71909-800c-4ca9-9edd-324209302a97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:53:06.171090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c71909-800c-4ca9-9edd-324209302a97 service neutron] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:06.171368 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c71909-800c-4ca9-9edd-324209302a97 service neutron] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:06.171641 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c71909-800c-4ca9-9edd-324209302a97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:06.173226 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-45c71909-800c-4ca9-9edd-324209302a97 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:06.270957 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:06.287980 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c71909-800c-4ca9-9edd-324209302a97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:53:06.289422 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c71909-800c-4ca9-9edd-324209302a97 service neutron] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:53:06.307380 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 675/1343] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:53:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:53:09.307549 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ab8556-124c-4ede-bedd-539e6185f8aa None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:09.308074 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ab8556-124c-4ede-bedd-539e6185f8aa None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:09.308278 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ab8556-124c-4ede-bedd-539e6185f8aa None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:09.309632 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 669/1344] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:53:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:09.314868 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0f6e0f-e1e0-4e1b-b1ab-454907ce8786 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:09.315272 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0f6e0f-e1e0-4e1b-b1ab-454907ce8786 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:09.315423 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0f6e0f-e1e0-4e1b-b1ab-454907ce8786 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:09.388643 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-2c0f6e0f-e1e0-4e1b-b1ab-454907ce8786 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:09.389859 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-2c0f6e0f-e1e0-4e1b-b1ab-454907ce8786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:09.489772 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 676/1345] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:53:09 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:09.782040 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ab8556-124c-4ede-bedd-539e6185f8aa None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:53:09.783615 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-a9ab8556-124c-4ede-bedd-539e6185f8aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:09.878659 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:09.898401 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ab8556-124c-4ede-bedd-539e6185f8aa None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:53:09.974461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65515ac1-4d32-45d5-8c82-46636d7cbd84 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:53:09.975070 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65515ac1-4d32-45d5-8c82-46636d7cbd84 service neutron] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:09.975352 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65515ac1-4d32-45d5-8c82-46636d7cbd84 service neutron] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:09.975628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65515ac1-4d32-45d5-8c82-46636d7cbd84 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:09.977219 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-65515ac1-4d32-45d5-8c82-46636d7cbd84 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:10.108665 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65515ac1-4d32-45d5-8c82-46636d7cbd84 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:53:10.110240 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65515ac1-4d32-45d5-8c82-46636d7cbd84 service neutron] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:53:10.130027 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 670/1346] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:53:09 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:53:14.474837 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edc38f0-a262-484e-9760-443c4b2c44ba None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:14.475429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edc38f0-a262-484e-9760-443c4b2c44ba None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:14.475429 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edc38f0-a262-484e-9760-443c4b2c44ba None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:14.480671 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 677/1347] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:53:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:14.483150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58b9643-b425-4fc4-8b68-326e48410a2b None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:14.483699 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58b9643-b425-4fc4-8b68-326e48410a2b None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:14.483978 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58b9643-b425-4fc4-8b68-326e48410a2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:14.503039 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:14.560773 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-e58b9643-b425-4fc4-8b68-326e48410a2b None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:14.562005 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-e58b9643-b425-4fc4-8b68-326e48410a2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:14.664946 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 671/1348] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:53:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:14.947598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edc38f0-a262-484e-9760-443c4b2c44ba None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:53:14.949132 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-3edc38f0-a262-484e-9760-443c4b2c44ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:15.007275 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:15.053874 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edc38f0-a262-484e-9760-443c4b2c44ba None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:53:15.127140 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7f5935-cad3-4393-a7f9-c7c5ecff09be service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:53:15.127750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7f5935-cad3-4393-a7f9-c7c5ecff09be service neutron] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:15.128031 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7f5935-cad3-4393-a7f9-c7c5ecff09be service neutron] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:15.128306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7f5935-cad3-4393-a7f9-c7c5ecff09be service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:15.130545 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-9d7f5935-cad3-4393-a7f9-c7c5ecff09be service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:15.235104 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:15.238836 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7f5935-cad3-4393-a7f9-c7c5ecff09be service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:53:15.241310 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7f5935-cad3-4393-a7f9-c7c5ecff09be service neutron] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:53:15.259903 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 678/1349] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:53:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:53:20.228560 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1646076-5dd5-43a0-87ce-5caa112a889b None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:20.228560 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1646076-5dd5-43a0-87ce-5caa112a889b None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:20.229014 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1646076-5dd5-43a0-87ce-5caa112a889b None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:20.230007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 672/1350] 10.222.0.27 () {54 vars in 921 bytes} [Wed Sep 7 03:53:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:20.235148 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc407ee2-2ac2-4cb9-8537-02febfe8c627 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:20.235569 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc407ee2-2ac2-4cb9-8537-02febfe8c627 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:20.235758 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc407ee2-2ac2-4cb9-8537-02febfe8c627 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:20.372994 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-dc407ee2-2ac2-4cb9-8537-02febfe8c627 None None] MFA Rules not processed for user `3bc3872e11a749758665f2c5dc1db8e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:20.374495 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-dc407ee2-2ac2-4cb9-8537-02febfe8c627 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:20.510770 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 679/1351] 10.222.0.27 () {58 vars in 1018 bytes} [Wed Sep 7 03:53:20 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:24.662900 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=59 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:24.745493 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7444bb-e8dd-4fe9-9db7-3d23010fa981 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:24.745847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7444bb-e8dd-4fe9-9db7-3d23010fa981 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:24.746043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7444bb-e8dd-4fe9-9db7-3d23010fa981 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:24.747582 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 673/1352] 10.222.0.27 () {54 vars in 921 bytes} [Wed Sep 7 03:53:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:24.752302 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f1b63b-300a-4338-990a-b9d5495b81f2 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:24.752734 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f1b63b-300a-4338-990a-b9d5495b81f2 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:24.752952 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f1b63b-300a-4338-990a-b9d5495b81f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:24.807807 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:24.833515 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-d6f1b63b-300a-4338-990a-b9d5495b81f2 None None] MFA Rules not processed for user `3bc3872e11a749758665f2c5dc1db8e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:24.834954 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-d6f1b63b-300a-4338-990a-b9d5495b81f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:24.957255 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 680/1353] 10.222.0.27 () {58 vars in 1018 bytes} [Wed Sep 7 03:53:24 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:26.197983 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bc22ed-6473-4964-b4bf-d812309c3123 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:26.198528 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bc22ed-6473-4964-b4bf-d812309c3123 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:26.198644 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bc22ed-6473-4964-b4bf-d812309c3123 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:26.199901 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 674/1354] 10.222.0.27 () {54 vars in 921 bytes} [Wed Sep 7 03:53:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:26.205077 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14058cb9-9a8d-4d98-824e-b071aeeeeb80 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:26.205490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14058cb9-9a8d-4d98-824e-b071aeeeeb80 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:26.205963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14058cb9-9a8d-4d98-824e-b071aeeeeb80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:26.290568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-14058cb9-9a8d-4d98-824e-b071aeeeeb80 None None] MFA Rules not processed for user `3bc3872e11a749758665f2c5dc1db8e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:26.291741 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-14058cb9-9a8d-4d98-824e-b071aeeeeb80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:26.333981 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:26.399180 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 681/1355] 10.222.0.27 () {58 vars in 1018 bytes} [Wed Sep 7 03:53:26 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:29.982804 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac9b6fc-d3ad-46f9-9009-0b50a713c847 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:29.983356 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac9b6fc-d3ad-46f9-9009-0b50a713c847 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:29.983356 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac9b6fc-d3ad-46f9-9009-0b50a713c847 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:29.984851 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 675/1356] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:53:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:29.990999 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae5a22e-f89c-460d-a4f6-255cb1bc478c None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:29.991853 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae5a22e-f89c-460d-a4f6-255cb1bc478c None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:29.992321 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae5a22e-f89c-460d-a4f6-255cb1bc478c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:30.063323 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-cae5a22e-f89c-460d-a4f6-255cb1bc478c None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:30.064748 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-cae5a22e-f89c-460d-a4f6-255cb1bc478c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:30.165134 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 682/1357] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:53:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:30.469321 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618f567e-9e82-4180-a6fa-82850e1f7182 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:30.469824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618f567e-9e82-4180-a6fa-82850e1f7182 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:30.470308 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618f567e-9e82-4180-a6fa-82850e1f7182 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:30.473264 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 676/1358] 10.222.0.27 () {54 vars in 962 bytes} [Wed Sep 7 03:53:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:30.480582 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ca4544-5e8c-4afb-890a-bf9770ee51a7 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:30.480983 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ca4544-5e8c-4afb-890a-bf9770ee51a7 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:30.481362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ca4544-5e8c-4afb-890a-bf9770ee51a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:30.507309 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:30.581428 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-10ca4544-5e8c-4afb-890a-bf9770ee51a7 None None] MFA Rules not processed for user `953cb6c24bcc4551b91ce540d18cd751`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:30.583945 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-10ca4544-5e8c-4afb-890a-bf9770ee51a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:30.706764 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 683/1359] 10.222.0.27 () {58 vars in 1059 bytes} [Wed Sep 7 03:53:30 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:30.715431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618f567e-9e82-4180-a6fa-82850e1f7182 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:53:30.717075 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-618f567e-9e82-4180-a6fa-82850e1f7182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:30.883333 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618f567e-9e82-4180-a6fa-82850e1f7182 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:53:30.960946 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:31.003176 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d081b6b4-1819-46ee-a0ce-0ca16baa8345 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:53:31.003949 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d081b6b4-1819-46ee-a0ce-0ca16baa8345 service nova] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:31.004397 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d081b6b4-1819-46ee-a0ce-0ca16baa8345 service nova] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:31.004940 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d081b6b4-1819-46ee-a0ce-0ca16baa8345 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:31.006695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-d081b6b4-1819-46ee-a0ce-0ca16baa8345 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:31.104800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d081b6b4-1819-46ee-a0ce-0ca16baa8345 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:53:31.106419 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d081b6b4-1819-46ee-a0ce-0ca16baa8345 service nova] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:53:31.125076 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 677/1360] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:53:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:53:33.506940 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d56e332-0e7a-4b50-ad7c-8eda57700c7d None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:33.507508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d56e332-0e7a-4b50-ad7c-8eda57700c7d None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:33.507508 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d56e332-0e7a-4b50-ad7c-8eda57700c7d None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:33.509004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 684/1361] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:53:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:33.514237 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57176cc-a954-45ac-9579-b4f2051b5f2a None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:33.514670 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57176cc-a954-45ac-9579-b4f2051b5f2a None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:33.514800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57176cc-a954-45ac-9579-b4f2051b5f2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:33.587173 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-c57176cc-a954-45ac-9579-b4f2051b5f2a None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:33.588597 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-c57176cc-a954-45ac-9579-b4f2051b5f2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:33.627320 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:33.693854 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 678/1362] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:53:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:33.993469 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d56e332-0e7a-4b50-ad7c-8eda57700c7d None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:53:33.995039 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-1d56e332-0e7a-4b50-ad7c-8eda57700c7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:34.075312 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:34.178484 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d56e332-0e7a-4b50-ad7c-8eda57700c7d None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:53:34.315528 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03589688-c69b-4a72-a1fa-51778faa6d7a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:53:34.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03589688-c69b-4a72-a1fa-51778faa6d7a service nova] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:34.316113 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03589688-c69b-4a72-a1fa-51778faa6d7a service nova] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:34.318148 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03589688-c69b-4a72-a1fa-51778faa6d7a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:34.318148 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-03589688-c69b-4a72-a1fa-51778faa6d7a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:34.342535 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:34.414583 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03589688-c69b-4a72-a1fa-51778faa6d7a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:53:34.416209 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03589688-c69b-4a72-a1fa-51778faa6d7a service nova] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:53:34.435668 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 685/1363] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:53:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:53:36.937456 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9e1a32-9296-479e-b1a7-e2281d3f4a98 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:36.937974 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9e1a32-9296-479e-b1a7-e2281d3f4a98 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:36.937974 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9e1a32-9296-479e-b1a7-e2281d3f4a98 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:36.939392 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 679/1364] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:53:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:36.945307 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13808ab7-f463-4453-83a1-e09bd5f700e0 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:36.945760 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13808ab7-f463-4453-83a1-e09bd5f700e0 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:36.945961 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13808ab7-f463-4453-83a1-e09bd5f700e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:37.015614 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-13808ab7-f463-4453-83a1-e09bd5f700e0 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:37.017291 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-13808ab7-f463-4453-83a1-e09bd5f700e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:37.119668 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:37.127584 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 686/1365] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:53:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:37.420904 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e91673b-b4db-4823-badc-27b653aa765e None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:37.421183 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e91673b-b4db-4823-badc-27b653aa765e None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:37.421693 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e91673b-b4db-4823-badc-27b653aa765e None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:37.423117 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 680/1366] 10.222.0.27 () {54 vars in 962 bytes} [Wed Sep 7 03:53:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:37.429313 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04df2d1e-72ab-43fa-bbe8-c9d064019665 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:37.429675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04df2d1e-72ab-43fa-bbe8-c9d064019665 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:37.430024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04df2d1e-72ab-43fa-bbe8-c9d064019665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:37.513060 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-04df2d1e-72ab-43fa-bbe8-c9d064019665 None None] MFA Rules not processed for user `953cb6c24bcc4551b91ce540d18cd751`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:37.514216 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-04df2d1e-72ab-43fa-bbe8-c9d064019665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:37.644014 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 687/1367] 10.222.0.27 () {58 vars in 1059 bytes} [Wed Sep 7 03:53:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:37.652574 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e91673b-b4db-4823-badc-27b653aa765e None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:53:37.654361 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-4e91673b-b4db-4823-badc-27b653aa765e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:37.806975 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e91673b-b4db-4823-badc-27b653aa765e None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:53:37.827915 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:37.933578 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e757c2b2-5402-4a9c-9258-767bc7d0053c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:53:37.934227 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e757c2b2-5402-4a9c-9258-767bc7d0053c service nova] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:37.934514 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e757c2b2-5402-4a9c-9258-767bc7d0053c service nova] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:37.934791 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e757c2b2-5402-4a9c-9258-767bc7d0053c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:37.936492 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-e757c2b2-5402-4a9c-9258-767bc7d0053c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:38.039192 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e757c2b2-5402-4a9c-9258-767bc7d0053c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:53:38.040740 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e757c2b2-5402-4a9c-9258-767bc7d0053c service nova] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:53:38.061263 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 681/1368] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:53:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 07 03:53:38.245693 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c828262c-88df-4278-a974-29ba2bbadcab None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:38.246377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c828262c-88df-4278-a974-29ba2bbadcab None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:38.246377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c828262c-88df-4278-a974-29ba2bbadcab None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:38.247683 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 688/1369] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:53:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:38.253776 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b84a05d-bee4-4986-9d3a-b09e0b22c597 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:38.258476 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b84a05d-bee4-4986-9d3a-b09e0b22c597 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:38.258476 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b84a05d-bee4-4986-9d3a-b09e0b22c597 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:38.303968 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:38.355638 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-2b84a05d-bee4-4986-9d3a-b09e0b22c597 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:38.358178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-2b84a05d-bee4-4986-9d3a-b09e0b22c597 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:38.480452 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 682/1370] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:53:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:38.875145 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7a615f-9d61-4aee-bf80-a42ba2df8aa2 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:38.875548 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7a615f-9d61-4aee-bf80-a42ba2df8aa2 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:38.875830 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7a615f-9d61-4aee-bf80-a42ba2df8aa2 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:38.881511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 689/1371] 10.222.0.27 () {54 vars in 964 bytes} [Wed Sep 7 03:53:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:38.885365 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7387ab58-c493-430e-8a64-e916134737e8 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:38.885731 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7387ab58-c493-430e-8a64-e916134737e8 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:38.886076 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7387ab58-c493-430e-8a64-e916134737e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:38.970884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-7387ab58-c493-430e-8a64-e916134737e8 None None] MFA Rules not processed for user `989081d67c8945dc8d853bb2769c902a`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:38.972177 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-7387ab58-c493-430e-8a64-e916134737e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:39.023560 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:39.057929 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 683/1372] 10.222.0.27 () {58 vars in 1061 bytes} [Wed Sep 7 03:53:38 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:39.066068 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7a615f-9d61-4aee-bf80-a42ba2df8aa2 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:53:39.067848 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-1d7a615f-9d61-4aee-bf80-a42ba2df8aa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:39.125210 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:39.183060 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7a615f-9d61-4aee-bf80-a42ba2df8aa2 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:53:39.286178 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdacf65f-d339-4439-82c3-bc4da2d8d337 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '989081d67c8945dc8d853bb2769c902a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:53:39.286178 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdacf65f-d339-4439-82c3-bc4da2d8d337 service cinder] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:39.286178 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdacf65f-d339-4439-82c3-bc4da2d8d337 service cinder] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:39.286178 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdacf65f-d339-4439-82c3-bc4da2d8d337 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:39.288619 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-cdacf65f-d339-4439-82c3-bc4da2d8d337 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:39.421891 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:39.441152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdacf65f-d339-4439-82c3-bc4da2d8d337 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:53:39.443582 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdacf65f-d339-4439-82c3-bc4da2d8d337 service cinder] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:53:39.471384 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 690/1373] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:53:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:53:40.385509 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eda82e-3772-4c42-8420-a46387b51e99 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:40.386072 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eda82e-3772-4c42-8420-a46387b51e99 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:40.386072 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eda82e-3772-4c42-8420-a46387b51e99 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:40.387431 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 684/1374] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:53:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:40.392635 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2551ae2-f464-49fd-83b5-e80ecb229946 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:40.393073 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2551ae2-f464-49fd-83b5-e80ecb229946 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:40.393178 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2551ae2-f464-49fd-83b5-e80ecb229946 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:40.459801 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-c2551ae2-f464-49fd-83b5-e80ecb229946 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:40.461025 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-c2551ae2-f464-49fd-83b5-e80ecb229946 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:40.560981 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 691/1375] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:53:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:40.852707 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07eda82e-3772-4c42-8420-a46387b51e99 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:53:40.854255 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-07eda82e-3772-4c42-8420-a46387b51e99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:41.012963 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07eda82e-3772-4c42-8420-a46387b51e99 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:53:41.070360 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:41.137761 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222f704c-41cc-4951-9dba-316d59d96f6c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:53:41.139240 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222f704c-41cc-4951-9dba-316d59d96f6c service nova] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:41.139240 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222f704c-41cc-4951-9dba-316d59d96f6c service nova] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:41.139436 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222f704c-41cc-4951-9dba-316d59d96f6c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:41.141510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-222f704c-41cc-4951-9dba-316d59d96f6c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:41.236413 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222f704c-41cc-4951-9dba-316d59d96f6c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:53:41.237970 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222f704c-41cc-4951-9dba-316d59d96f6c service nova] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:53:41.257598 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 685/1376] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:53:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:53:43.493749 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f1dd80-e375-48c3-8eb9-e6a4f1a30b91 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:43.494304 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f1dd80-e375-48c3-8eb9-e6a4f1a30b91 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:43.494417 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f1dd80-e375-48c3-8eb9-e6a4f1a30b91 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:43.495855 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 692/1377] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:53:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:43.501007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdb403b-0564-4e45-b684-6b7a07c93742 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:43.501440 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdb403b-0564-4e45-b684-6b7a07c93742 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:43.501611 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdb403b-0564-4e45-b684-6b7a07c93742 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:43.572242 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-7cdb403b-0564-4e45-b684-6b7a07c93742 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:43.573378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-7cdb403b-0564-4e45-b684-6b7a07c93742 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:43.580795 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:43.680751 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 686/1378] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:53:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:43.971068 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f1dd80-e375-48c3-8eb9-e6a4f1a30b91 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:53:43.972834 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-88f1dd80-e375-48c3-8eb9-e6a4f1a30b91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:44.037088 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:44.131856 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f1dd80-e375-48c3-8eb9-e6a4f1a30b91 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:53:44.252126 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fd4046-b840-47ea-88f5-992cc10a0648 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:53:44.252925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fd4046-b840-47ea-88f5-992cc10a0648 service nova] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:44.253329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fd4046-b840-47ea-88f5-992cc10a0648 service nova] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:44.257154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fd4046-b840-47ea-88f5-992cc10a0648 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:44.257154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-96fd4046-b840-47ea-88f5-992cc10a0648 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:44.277454 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:44.352461 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fd4046-b840-47ea-88f5-992cc10a0648 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:53:44.354055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fd4046-b840-47ea-88f5-992cc10a0648 service nova] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:53:44.373520 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 693/1379] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:53:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:53:46.619004 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa6cb41-3c0f-4287-947f-ee4833653a91 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:46.619579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa6cb41-3c0f-4287-947f-ee4833653a91 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:46.619579 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa6cb41-3c0f-4287-947f-ee4833653a91 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:46.624922 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 687/1380] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:53:46 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:46.626376 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674bb596-eb30-4bbd-96f3-5de8176c2177 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:46.626811 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674bb596-eb30-4bbd-96f3-5de8176c2177 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:46.626921 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674bb596-eb30-4bbd-96f3-5de8176c2177 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:46.702406 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-674bb596-eb30-4bbd-96f3-5de8176c2177 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:46.703868 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-674bb596-eb30-4bbd-96f3-5de8176c2177 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:46.816745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:46.825430 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 694/1381] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:53:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:47.115378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa6cb41-3c0f-4287-947f-ee4833653a91 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:53:47.116921 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-dfa6cb41-3c0f-4287-947f-ee4833653a91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:47.261628 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:47.273551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa6cb41-3c0f-4287-947f-ee4833653a91 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:53:47.387732 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9023a46b-5519-4d86-8247-5bd245407cf3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:53:47.388340 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9023a46b-5519-4d86-8247-5bd245407cf3 service nova] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:47.388650 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9023a46b-5519-4d86-8247-5bd245407cf3 service nova] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:47.388940 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9023a46b-5519-4d86-8247-5bd245407cf3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:47.390596 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-9023a46b-5519-4d86-8247-5bd245407cf3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:47.483737 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9023a46b-5519-4d86-8247-5bd245407cf3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:53:47.485598 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9023a46b-5519-4d86-8247-5bd245407cf3 service nova] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:53:47.496712 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:47.506457 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 688/1382] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:53:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:53:49.873353 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9a5752-634f-45a8-8aba-f9a1850f7e6e None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:49.873962 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9a5752-634f-45a8-8aba-f9a1850f7e6e None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:49.874089 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9a5752-634f-45a8-8aba-f9a1850f7e6e None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:49.875491 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 695/1383] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:53:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:49.881771 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87af3d70-8f77-4c28-9023-f318ccd7dab8 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:49.882223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87af3d70-8f77-4c28-9023-f318ccd7dab8 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:49.882388 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87af3d70-8f77-4c28-9023-f318ccd7dab8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:49.951895 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-87af3d70-8f77-4c28-9023-f318ccd7dab8 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:49.953066 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-87af3d70-8f77-4c28-9023-f318ccd7dab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:50.095650 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 689/1384] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:53:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:50.398387 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9a5752-634f-45a8-8aba-f9a1850f7e6e None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:53:50.399975 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-3e9a5752-634f-45a8-8aba-f9a1850f7e6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:50.556059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9a5752-634f-45a8-8aba-f9a1850f7e6e None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:53:50.628548 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:50.678032 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad22964-a016-4f8d-b5cb-3d9ade313a15 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:53:50.678703 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad22964-a016-4f8d-b5cb-3d9ade313a15 service nova] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:50.678995 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad22964-a016-4f8d-b5cb-3d9ade313a15 service nova] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:50.679265 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad22964-a016-4f8d-b5cb-3d9ade313a15 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:50.680867 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-dad22964-a016-4f8d-b5cb-3d9ade313a15 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:50.773642 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad22964-a016-4f8d-b5cb-3d9ade313a15 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:53:50.775361 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad22964-a016-4f8d-b5cb-3d9ade313a15 service nova] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:53:50.794515 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 696/1385] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:53:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:53:53.006238 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2015813a-66c5-4d3f-b5dd-6ddf19c7c7a0 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:53.006801 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2015813a-66c5-4d3f-b5dd-6ddf19c7c7a0 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:53.006931 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2015813a-66c5-4d3f-b5dd-6ddf19c7c7a0 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:53.008739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 690/1386] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:53:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:53.014363 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940c6a42-1319-46b4-85bc-35f3fe6fe913 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:53.014926 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940c6a42-1319-46b4-85bc-35f3fe6fe913 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:53.014926 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940c6a42-1319-46b4-85bc-35f3fe6fe913 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:53.089172 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-940c6a42-1319-46b4-85bc-35f3fe6fe913 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:53.090466 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-940c6a42-1319-46b4-85bc-35f3fe6fe913 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:53.125581 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:53.204863 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 697/1387] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:53:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:53.500779 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2015813a-66c5-4d3f-b5dd-6ddf19c7c7a0 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:53:53.502352 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-2015813a-66c5-4d3f-b5dd-6ddf19c7c7a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:53.582445 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:53.657392 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2015813a-66c5-4d3f-b5dd-6ddf19c7c7a0 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:53:53.783738 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2817bbce-e2bd-438f-98a1-8af365555c5f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:53:53.784408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2817bbce-e2bd-438f-98a1-8af365555c5f service nova] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:53.784721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2817bbce-e2bd-438f-98a1-8af365555c5f service nova] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:53.785560 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2817bbce-e2bd-438f-98a1-8af365555c5f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:53.787457 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-2817bbce-e2bd-438f-98a1-8af365555c5f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:53.828461 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:53.896310 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2817bbce-e2bd-438f-98a1-8af365555c5f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:53:53.898794 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2817bbce-e2bd-438f-98a1-8af365555c5f service nova] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:53:53.922931 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 691/1388] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:53:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:53:56.457597 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03745c6-4b0c-4b1a-a5fe-a59ac9315c8b None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:56.458473 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03745c6-4b0c-4b1a-a5fe-a59ac9315c8b None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:56.458645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03745c6-4b0c-4b1a-a5fe-a59ac9315c8b None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:56.459956 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 698/1389] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:53:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:56.465692 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147eff48-d785-4de2-a892-164313e5e6d7 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:56.466208 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147eff48-d785-4de2-a892-164313e5e6d7 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:56.466369 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147eff48-d785-4de2-a892-164313e5e6d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:56.533776 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-147eff48-d785-4de2-a892-164313e5e6d7 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:56.535055 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-147eff48-d785-4de2-a892-164313e5e6d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:56.639609 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 692/1390] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:53:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:56.940245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03745c6-4b0c-4b1a-a5fe-a59ac9315c8b None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:53:56.942349 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-b03745c6-4b0c-4b1a-a5fe-a59ac9315c8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:57.091395 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03745c6-4b0c-4b1a-a5fe-a59ac9315c8b None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:53:57.107716 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:57.205367 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecadb13c-2cd1-4870-98ff-c019c87eebbb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:53:57.205901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecadb13c-2cd1-4870-98ff-c019c87eebbb service nova] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:57.206215 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecadb13c-2cd1-4870-98ff-c019c87eebbb service nova] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:57.206503 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecadb13c-2cd1-4870-98ff-c019c87eebbb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:57.209005 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-ecadb13c-2cd1-4870-98ff-c019c87eebbb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:57.307205 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecadb13c-2cd1-4870-98ff-c019c87eebbb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:53:57.308757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecadb13c-2cd1-4870-98ff-c019c87eebbb service nova] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:53:57.327439 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 699/1391] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:53:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:53:58.726178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf1f6a7-61df-4d34-a018-9cad9b8f9ea1 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:58.726804 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf1f6a7-61df-4d34-a018-9cad9b8f9ea1 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:58.726804 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf1f6a7-61df-4d34-a018-9cad9b8f9ea1 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:58.728274 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 693/1392] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:53:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:58.733418 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e2dde9-1b7c-47ac-9f7a-bef97551cc43 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:58.733843 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e2dde9-1b7c-47ac-9f7a-bef97551cc43 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:58.734103 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e2dde9-1b7c-47ac-9f7a-bef97551cc43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:58.761892 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:58.804404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-67e2dde9-1b7c-47ac-9f7a-bef97551cc43 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:58.805630 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-67e2dde9-1b7c-47ac-9f7a-bef97551cc43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:58.904268 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 700/1393] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:53:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:53:59.207903 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950e0e54-18f0-4ef9-9936-77a3b9878399 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:59.208188 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950e0e54-18f0-4ef9-9936-77a3b9878399 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:59.208489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950e0e54-18f0-4ef9-9936-77a3b9878399 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:59.209966 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 694/1394] 10.222.0.27 () {54 vars in 961 bytes} [Wed Sep 7 03:53:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:59.215989 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121adcd0-ccfd-4b51-8109-d50d7f596100 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:59.216283 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121adcd0-ccfd-4b51-8109-d50d7f596100 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:59.216565 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121adcd0-ccfd-4b51-8109-d50d7f596100 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:59.299714 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-121adcd0-ccfd-4b51-8109-d50d7f596100 None None] MFA Rules not processed for user `dfa09be69ee3444e94f00cc139bf50e7`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:59.300963 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-121adcd0-ccfd-4b51-8109-d50d7f596100 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:59.343779 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:59.381446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 701/1395] 10.222.0.27 () {58 vars in 1058 bytes} [Wed Sep 7 03:53:59 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 07 03:53:59.391169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950e0e54-18f0-4ef9-9936-77a3b9878399 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:53:59.392835 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-950e0e54-18f0-4ef9-9936-77a3b9878399 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:59.408539 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:59.507512 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950e0e54-18f0-4ef9-9936-77a3b9878399 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:53:59.517948 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d25bae-54de-4ffd-b9aa-143af10213ee None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:59.518258 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d25bae-54de-4ffd-b9aa-143af10213ee None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:59.518531 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d25bae-54de-4ffd-b9aa-143af10213ee None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:59.520376 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 702/1396] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:53:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:53:59.525383 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bd2cce-2f55-419e-ab8a-e6dc40d2d2b3 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:59.525670 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bd2cce-2f55-419e-ab8a-e6dc40d2d2b3 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:59.525941 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bd2cce-2f55-419e-ab8a-e6dc40d2d2b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:59.583664 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0c1944-583b-4ae2-b99c-d9242deab9ba service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dfa09be69ee3444e94f00cc139bf50e7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:53:59.584269 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0c1944-583b-4ae2-b99c-d9242deab9ba service glance] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:53:59.584547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0c1944-583b-4ae2-b99c-d9242deab9ba service glance] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:53:59.584814 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0c1944-583b-4ae2-b99c-d9242deab9ba service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:53:59.586568 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-ec0c1944-583b-4ae2-b99c-d9242deab9ba service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:59.595196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-c1bd2cce-2f55-419e-ab8a-e6dc40d2d2b3 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:53:59.597801 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-c1bd2cce-2f55-419e-ab8a-e6dc40d2d2b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:53:59.663603 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:53:59.704169 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 703/1397] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:53:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 07 03:53:59.708232 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0c1944-583b-4ae2-b99c-d9242deab9ba service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:53:59.709671 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0c1944-583b-4ae2-b99c-d9242deab9ba service glance] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:53:59.727641 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 695/1398] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:53:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:53:59.994396 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bd2cce-2f55-419e-ab8a-e6dc40d2d2b3 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:53:59.995594 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-c1bd2cce-2f55-419e-ab8a-e6dc40d2d2b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:00.044288 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:00.113978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bd2cce-2f55-419e-ab8a-e6dc40d2d2b3 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:00.237346 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b492e4-e7b8-4a20-bd21-b765c52f8925 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:00.237346 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b492e4-e7b8-4a20-bd21-b765c52f8925 service nova] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:00.237346 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b492e4-e7b8-4a20-bd21-b765c52f8925 service nova] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:00.237346 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b492e4-e7b8-4a20-bd21-b765c52f8925 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:00.238354 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-e3b492e4-e7b8-4a20-bd21-b765c52f8925 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:00.283252 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:00.337322 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b492e4-e7b8-4a20-bd21-b765c52f8925 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:00.339072 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b492e4-e7b8-4a20-bd21-b765c52f8925 service nova] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:00.359315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 704/1399] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:53:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:54:00.482900 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19154130-4b12-4f92-9ee9-37932c0cdcf6 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:00.483083 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19154130-4b12-4f92-9ee9-37932c0cdcf6 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:00.483353 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19154130-4b12-4f92-9ee9-37932c0cdcf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:00.568781 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-19154130-4b12-4f92-9ee9-37932c0cdcf6 None None] MFA Rules not processed for user `e7b8f511ee884dbfb4f56af78bf75984`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:00.570072 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-19154130-4b12-4f92-9ee9-37932c0cdcf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:00.663149 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 696/1400] 10.222.0.27 () {58 vars in 1009 bytes} [Wed Sep 7 03:54:00 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:00.711411 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29db1f95-6d90-4e46-a113-a880d30c7f25 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:00.711698 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29db1f95-6d90-4e46-a113-a880d30c7f25 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:00.711978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29db1f95-6d90-4e46-a113-a880d30c7f25 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:00.713360 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 705/1401] 10.222.0.27 () {54 vars in 943 bytes} [Wed Sep 7 03:54:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:00.719606 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbab120-fd2e-47cc-a2fb-ef7e794e4b3d None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:00.719896 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbab120-fd2e-47cc-a2fb-ef7e794e4b3d None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:00.720195 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbab120-fd2e-47cc-a2fb-ef7e794e4b3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:00.763191 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:00.805606 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-2fbab120-fd2e-47cc-a2fb-ef7e794e4b3d None None] MFA Rules not processed for user `f5c3ab9728f74cd8bb2614bcca9e8619`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:00.806864 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-2fbab120-fd2e-47cc-a2fb-ef7e794e4b3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:00.915118 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 697/1402] 10.222.0.27 () {58 vars in 1040 bytes} [Wed Sep 7 03:54:00 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:00.924647 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29db1f95-6d90-4e46-a113-a880d30c7f25 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:00.927223 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-29db1f95-6d90-4e46-a113-a880d30c7f25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:01.063795 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29db1f95-6d90-4e46-a113-a880d30c7f25 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:01.096860 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:01.173130 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876bc5a6-8126-47e0-95c4-43661d620461 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f5c3ab9728f74cd8bb2614bcca9e8619', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:01.173746 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876bc5a6-8126-47e0-95c4-43661d620461 service swift] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:01.174026 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876bc5a6-8126-47e0-95c4-43661d620461 service swift] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:01.174327 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876bc5a6-8126-47e0-95c4-43661d620461 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:01.176704 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-876bc5a6-8126-47e0-95c4-43661d620461 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:01.276869 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876bc5a6-8126-47e0-95c4-43661d620461 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:01.278789 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876bc5a6-8126-47e0-95c4-43661d620461 service swift] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:01.288346 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 706/1403] 10.222.0.27 () {60 vars in 1384 bytes} [Wed Sep 7 03:54:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 07 03:54:02.610281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73771628-a344-43d4-bcd0-1fb4d7034413 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:02.610870 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73771628-a344-43d4-bcd0-1fb4d7034413 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:02.610870 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73771628-a344-43d4-bcd0-1fb4d7034413 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:02.612078 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 698/1404] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:54:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:02.618446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d5b6f0-f1e9-4e78-90ff-9d2a8352eed8 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:02.619317 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d5b6f0-f1e9-4e78-90ff-9d2a8352eed8 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:02.619888 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d5b6f0-f1e9-4e78-90ff-9d2a8352eed8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:02.699751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-54d5b6f0-f1e9-4e78-90ff-9d2a8352eed8 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:02.699751 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-54d5b6f0-f1e9-4e78-90ff-9d2a8352eed8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:02.713922 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:02.810208 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 707/1405] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:54:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 07 03:54:03.101553 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73771628-a344-43d4-bcd0-1fb4d7034413 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:03.103236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-73771628-a344-43d4-bcd0-1fb4d7034413 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:03.199038 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:03.274500 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73771628-a344-43d4-bcd0-1fb4d7034413 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:03.403660 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8237510d-2e08-4456-b785-a6cc7a705166 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:03.404776 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8237510d-2e08-4456-b785-a6cc7a705166 service nova] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:03.405063 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8237510d-2e08-4456-b785-a6cc7a705166 service nova] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:03.405338 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8237510d-2e08-4456-b785-a6cc7a705166 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:03.407084 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-8237510d-2e08-4456-b785-a6cc7a705166 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:03.470550 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:03.512640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8237510d-2e08-4456-b785-a6cc7a705166 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:03.514377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8237510d-2e08-4456-b785-a6cc7a705166 service nova] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:03.537225 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 699/1406] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:54:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:54:05.732965 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f04f9e2-b50a-4594-b3dc-30b805fcef2e None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:05.733527 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f04f9e2-b50a-4594-b3dc-30b805fcef2e None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:05.733527 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f04f9e2-b50a-4594-b3dc-30b805fcef2e None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:05.735083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 708/1407] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:54:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:05.741128 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8028fc-afbc-490f-9cc4-bd4c15e3bb03 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:05.741584 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8028fc-afbc-490f-9cc4-bd4c15e3bb03 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:05.741677 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8028fc-afbc-490f-9cc4-bd4c15e3bb03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:05.814447 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-2a8028fc-afbc-490f-9cc4-bd4c15e3bb03 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:05.815755 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-2a8028fc-afbc-490f-9cc4-bd4c15e3bb03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:05.921784 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 700/1408] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:54:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:06.211421 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f04f9e2-b50a-4594-b3dc-30b805fcef2e None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:06.213001 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-8f04f9e2-b50a-4594-b3dc-30b805fcef2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:06.416260 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:06.421500 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f04f9e2-b50a-4594-b3dc-30b805fcef2e None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:06.589587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f088e5-d484-48c0-98c6-6aee98c38969 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:06.590365 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f088e5-d484-48c0-98c6-6aee98c38969 service nova] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:06.590675 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f088e5-d484-48c0-98c6-6aee98c38969 service nova] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:06.590950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f088e5-d484-48c0-98c6-6aee98c38969 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:06.594075 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-c9f088e5-d484-48c0-98c6-6aee98c38969 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:06.733924 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f088e5-d484-48c0-98c6-6aee98c38969 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:06.735523 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f088e5-d484-48c0-98c6-6aee98c38969 service nova] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:06.757167 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 709/1409] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:54:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 546 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:54:15.919684 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=86 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:16.752562 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:24.207706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c179cdb5-26da-4080-be96-b48f8078134f None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:24.208291 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c179cdb5-26da-4080-be96-b48f8078134f None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:24.208291 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c179cdb5-26da-4080-be96-b48f8078134f None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:24.209583 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 701/1410] 10.222.0.27 () {54 vars in 918 bytes} [Wed Sep 7 03:54:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 07 03:54:24.214738 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1067ae-6ac7-4362-8237-b059e324d607 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:24.215173 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1067ae-6ac7-4362-8237-b059e324d607 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:24.215270 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1067ae-6ac7-4362-8237-b059e324d607 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:24.297315 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-8d1067ae-6ac7-4362-8237-b059e324d607 None None] MFA Rules not processed for user `3bc3872e11a749758665f2c5dc1db8e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:24.298567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-8d1067ae-6ac7-4362-8237-b059e324d607 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:24.403848 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 710/1411] 10.222.0.27 () {58 vars in 1015 bytes} [Wed Sep 7 03:54:24 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:24.418487 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d266401-09fe-4088-8ac2-4fd3a0ceda5c None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:24.418800 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d266401-09fe-4088-8ac2-4fd3a0ceda5c None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:24.419084 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d266401-09fe-4088-8ac2-4fd3a0ceda5c None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:24.420416 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 702/1412] 10.222.0.27 () {54 vars in 918 bytes} [Wed Sep 7 03:54:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:24.425604 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf2da2b-a8ce-4068-848d-128c149ce665 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:24.425919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf2da2b-a8ce-4068-848d-128c149ce665 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:24.426245 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf2da2b-a8ce-4068-848d-128c149ce665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:24.498982 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:24.501774 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-1bf2da2b-a8ce-4068-848d-128c149ce665 None None] MFA Rules not processed for user `953cb6c24bcc4551b91ce540d18cd751`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:24.503781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-1bf2da2b-a8ce-4068-848d-128c149ce665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:24.619118 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 711/1413] 10.222.0.27 () {58 vars in 1015 bytes} [Wed Sep 7 03:54:24 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:27.210508 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d61172-b14b-46a1-83a4-14a2a4a96d26 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:27.211086 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d61172-b14b-46a1-83a4-14a2a4a96d26 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:27.211086 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d61172-b14b-46a1-83a4-14a2a4a96d26 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:27.212362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 703/1414] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 7 03:54:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:27.217672 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4904d11-bc95-4276-9ceb-77aed8111eb0 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:27.218101 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4904d11-bc95-4276-9ceb-77aed8111eb0 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:27.218243 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4904d11-bc95-4276-9ceb-77aed8111eb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:27.284840 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-b4904d11-bc95-4276-9ceb-77aed8111eb0 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:27.286057 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-b4904d11-bc95-4276-9ceb-77aed8111eb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:27.342622 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:27.384598 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 712/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 7 03:54:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:27.678099 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d61172-b14b-46a1-83a4-14a2a4a96d26 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:27.679688 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-33d61172-b14b-46a1-83a4-14a2a4a96d26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:27.824603 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d61172-b14b-46a1-83a4-14a2a4a96d26 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:27.896254 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:27.935778 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7a8692-c60f-45d2-8b5a-d3ce97f15767 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:27.936351 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7a8692-c60f-45d2-8b5a-d3ce97f15767 service nova] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:27.936633 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7a8692-c60f-45d2-8b5a-d3ce97f15767 service nova] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:27.936907 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7a8692-c60f-45d2-8b5a-d3ce97f15767 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:27.938552 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-fe7a8692-c60f-45d2-8b5a-d3ce97f15767 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:28.028138 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7a8692-c60f-45d2-8b5a-d3ce97f15767 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:28.029739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7a8692-c60f-45d2-8b5a-d3ce97f15767 service nova] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:28.047303 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 704/1416] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:54:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:54:29.198306 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b22a0c-1705-4f68-b830-9642c9e63170 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:29.198865 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b22a0c-1705-4f68-b830-9642c9e63170 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:29.198865 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b22a0c-1705-4f68-b830-9642c9e63170 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:29.200242 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 713/1417] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 7 03:54:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:29.206994 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135f2909-7b89-4465-bcb9-8c93aa8c2ebc None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:29.207729 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135f2909-7b89-4465-bcb9-8c93aa8c2ebc None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:29.207830 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135f2909-7b89-4465-bcb9-8c93aa8c2ebc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:29.284451 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-135f2909-7b89-4465-bcb9-8c93aa8c2ebc None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:29.285825 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-135f2909-7b89-4465-bcb9-8c93aa8c2ebc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:29.319269 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:29.389980 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 705/1418] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 7 03:54:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:29.541467 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b22a0c-1705-4f68-b830-9642c9e63170 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:29.542932 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-72b22a0c-1705-4f68-b830-9642c9e63170 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:29.694313 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b22a0c-1705-4f68-b830-9642c9e63170 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:29.735168 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:29.808734 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95567046-e9e5-42c1-b4eb-36696c4f3eba service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:29.809263 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95567046-e9e5-42c1-b4eb-36696c4f3eba service nova] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:29.809538 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95567046-e9e5-42c1-b4eb-36696c4f3eba service nova] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:29.809805 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95567046-e9e5-42c1-b4eb-36696c4f3eba service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:29.811418 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-95567046-e9e5-42c1-b4eb-36696c4f3eba service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:29.899373 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95567046-e9e5-42c1-b4eb-36696c4f3eba service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:29.901009 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95567046-e9e5-42c1-b4eb-36696c4f3eba service nova] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:29.919618 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 714/1419] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:54:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 07 03:54:31.055548 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9a4514-5a3e-4495-bbda-95a216d138ee None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:31.056084 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9a4514-5a3e-4495-bbda-95a216d138ee None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:31.056084 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9a4514-5a3e-4495-bbda-95a216d138ee None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:31.057390 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 706/1420] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 7 03:54:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:31.063837 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21259bf-a7e1-4f0c-bcd9-5c5baf28d1ff None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:31.064285 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21259bf-a7e1-4f0c-bcd9-5c5baf28d1ff None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:31.064404 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21259bf-a7e1-4f0c-bcd9-5c5baf28d1ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:31.119901 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:31.142850 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-d21259bf-a7e1-4f0c-bcd9-5c5baf28d1ff None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:31.144107 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-d21259bf-a7e1-4f0c-bcd9-5c5baf28d1ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:31.240555 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 715/1421] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 7 03:54:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:31.395513 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9a4514-5a3e-4495-bbda-95a216d138ee None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:31.397039 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-4c9a4514-5a3e-4495-bbda-95a216d138ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:31.540377 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9a4514-5a3e-4495-bbda-95a216d138ee None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:31.552421 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:31.653992 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bd78f8-281e-4652-b3f5-b1067f1ea923 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:31.654554 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bd78f8-281e-4652-b3f5-b1067f1ea923 service nova] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:31.654859 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bd78f8-281e-4652-b3f5-b1067f1ea923 service nova] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:31.655157 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bd78f8-281e-4652-b3f5-b1067f1ea923 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:31.656758 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-f6bd78f8-281e-4652-b3f5-b1067f1ea923 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:31.749736 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bd78f8-281e-4652-b3f5-b1067f1ea923 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:31.751383 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bd78f8-281e-4652-b3f5-b1067f1ea923 service nova] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:31.771665 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 707/1422] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:54:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:54:32.890980 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1dcbc6-3e65-444a-a5eb-8e8e3358d663 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:32.891587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1dcbc6-3e65-444a-a5eb-8e8e3358d663 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:32.891587 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1dcbc6-3e65-444a-a5eb-8e8e3358d663 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:32.893648 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 716/1423] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 7 03:54:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:32.899828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d286b80-2a74-4737-badf-73d330f71c64 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:32.900255 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d286b80-2a74-4737-badf-73d330f71c64 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:32.900459 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d286b80-2a74-4737-badf-73d330f71c64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:32.920251 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:32.977343 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-4d286b80-2a74-4737-badf-73d330f71c64 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:32.978593 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-4d286b80-2a74-4737-badf-73d330f71c64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:33.074713 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 708/1424] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 7 03:54:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:33.225361 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1dcbc6-3e65-444a-a5eb-8e8e3358d663 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:33.226825 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-2a1dcbc6-3e65-444a-a5eb-8e8e3358d663 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:33.335780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:33.373087 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1dcbc6-3e65-444a-a5eb-8e8e3358d663 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:33.484416 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf46df3-6150-49a3-899f-b7a6cce26202 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:33.484937 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf46df3-6150-49a3-899f-b7a6cce26202 service nova] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:33.485211 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf46df3-6150-49a3-899f-b7a6cce26202 service nova] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:33.485475 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf46df3-6150-49a3-899f-b7a6cce26202 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:33.488012 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-aaf46df3-6150-49a3-899f-b7a6cce26202 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:33.562279 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:33.577051 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf46df3-6150-49a3-899f-b7a6cce26202 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:33.578718 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf46df3-6150-49a3-899f-b7a6cce26202 service nova] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:33.597702 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 717/1425] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:54:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:54:34.711848 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dde79e-f41d-475b-b62d-6eda0905848a None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:34.712385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dde79e-f41d-475b-b62d-6eda0905848a None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:34.712510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dde79e-f41d-475b-b62d-6eda0905848a None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:34.713790 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 709/1426] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 7 03:54:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:34.720213 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af577e5-e833-4d6e-86b4-cecff5340a98 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:34.720645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af577e5-e833-4d6e-86b4-cecff5340a98 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:34.721048 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af577e5-e833-4d6e-86b4-cecff5340a98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:34.796123 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-2af577e5-e833-4d6e-86b4-cecff5340a98 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:34.797342 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-2af577e5-e833-4d6e-86b4-cecff5340a98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:34.891768 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 718/1427] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 7 03:54:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:35.043700 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99dde79e-f41d-475b-b62d-6eda0905848a None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:35.045152 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-99dde79e-f41d-475b-b62d-6eda0905848a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:35.129946 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:35.192299 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99dde79e-f41d-475b-b62d-6eda0905848a None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:35.302547 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a6b81c-ff60-44b3-b0cc-4b126e2f9d3e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:35.303092 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a6b81c-ff60-44b3-b0cc-4b126e2f9d3e service nova] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:35.303374 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a6b81c-ff60-44b3-b0cc-4b126e2f9d3e service nova] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:35.303646 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a6b81c-ff60-44b3-b0cc-4b126e2f9d3e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:35.305245 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-c4a6b81c-ff60-44b3-b0cc-4b126e2f9d3e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:35.356248 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:35.400061 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a6b81c-ff60-44b3-b0cc-4b126e2f9d3e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:35.401665 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a6b81c-ff60-44b3-b0cc-4b126e2f9d3e service nova] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:35.419921 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 710/1428] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:54:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:54:36.477823 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955f1a46-cbf1-4b4a-bac0-e62dd2a62590 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:36.478414 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955f1a46-cbf1-4b4a-bac0-e62dd2a62590 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:36.478414 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955f1a46-cbf1-4b4a-bac0-e62dd2a62590 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:36.479780 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 719/1429] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 7 03:54:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:36.485870 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885aaaba-84e6-4e4b-81f0-afbc3dd762f5 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:36.486902 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885aaaba-84e6-4e4b-81f0-afbc3dd762f5 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:36.487014 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885aaaba-84e6-4e4b-81f0-afbc3dd762f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:36.563388 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-885aaaba-84e6-4e4b-81f0-afbc3dd762f5 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:36.565200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-885aaaba-84e6-4e4b-81f0-afbc3dd762f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:36.662761 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 711/1430] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 7 03:54:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:36.812756 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955f1a46-cbf1-4b4a-bac0-e62dd2a62590 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:36.814227 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-955f1a46-cbf1-4b4a-bac0-e62dd2a62590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:36.863877 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:36.960584 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955f1a46-cbf1-4b4a-bac0-e62dd2a62590 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:37.076150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4c27b8-b8a3-4c8a-ba75-eeae7b383367 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:37.076783 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4c27b8-b8a3-4c8a-ba75-eeae7b383367 service nova] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:37.077045 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4c27b8-b8a3-4c8a-ba75-eeae7b383367 service nova] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:37.077316 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4c27b8-b8a3-4c8a-ba75-eeae7b383367 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:37.078950 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-fc4c27b8-b8a3-4c8a-ba75-eeae7b383367 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:37.086745 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:37.170651 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4c27b8-b8a3-4c8a-ba75-eeae7b383367 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:37.172271 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4c27b8-b8a3-4c8a-ba75-eeae7b383367 service nova] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:37.189813 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 720/1431] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:54:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:54:38.253281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d7b229-acb4-4ac7-8bdd-2288e97a8007 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:38.253871 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d7b229-acb4-4ac7-8bdd-2288e97a8007 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:38.253871 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d7b229-acb4-4ac7-8bdd-2288e97a8007 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:38.255255 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 712/1432] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 7 03:54:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:38.261741 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72546b62-b18a-4550-92e8-646d4e751963 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:38.262223 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72546b62-b18a-4550-92e8-646d4e751963 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:38.262339 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72546b62-b18a-4550-92e8-646d4e751963 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:38.337375 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-72546b62-b18a-4550-92e8-646d4e751963 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:38.338619 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-72546b62-b18a-4550-92e8-646d4e751963 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:38.405551 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:38.435083 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 721/1433] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 7 03:54:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:38.587281 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d7b229-acb4-4ac7-8bdd-2288e97a8007 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:38.589023 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-d2d7b229-acb4-4ac7-8bdd-2288e97a8007 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:38.607256 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:38.733497 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d7b229-acb4-4ac7-8bdd-2288e97a8007 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:38.820166 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:38.847944 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92aec599-1441-46df-8a52-7d4fcbc38735 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:38.848476 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92aec599-1441-46df-8a52-7d4fcbc38735 service nova] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:38.848756 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92aec599-1441-46df-8a52-7d4fcbc38735 service nova] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:38.849025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92aec599-1441-46df-8a52-7d4fcbc38735 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:38.850672 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-92aec599-1441-46df-8a52-7d4fcbc38735 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:38.941117 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92aec599-1441-46df-8a52-7d4fcbc38735 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:38.942751 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92aec599-1441-46df-8a52-7d4fcbc38735 service nova] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:38.961309 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 713/1434] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:54:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:54:40.126779 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edbfebe-d7a8-4c56-afcf-b4115b5f9ff1 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:40.127344 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edbfebe-d7a8-4c56-afcf-b4115b5f9ff1 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:40.127596 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edbfebe-d7a8-4c56-afcf-b4115b5f9ff1 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:40.128956 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 722/1435] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 7 03:54:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:40.135745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac756659-e02d-49ab-8b90-958f44cff1ee None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:40.136235 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac756659-e02d-49ab-8b90-958f44cff1ee None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:40.136346 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac756659-e02d-49ab-8b90-958f44cff1ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:40.212523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-ac756659-e02d-49ab-8b90-958f44cff1ee None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:40.213726 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-ac756659-e02d-49ab-8b90-958f44cff1ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:40.255025 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:40.311981 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 714/1436] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 7 03:54:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:40.474007 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edbfebe-d7a8-4c56-afcf-b4115b5f9ff1 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:40.475567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-9edbfebe-d7a8-4c56-afcf-b4115b5f9ff1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:40.624752 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edbfebe-d7a8-4c56-afcf-b4115b5f9ff1 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:40.677958 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:40.737872 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c9b397-ac52-4bb4-8964-f7a02dddba61 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:40.738426 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c9b397-ac52-4bb4-8964-f7a02dddba61 service nova] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:40.738705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c9b397-ac52-4bb4-8964-f7a02dddba61 service nova] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:40.738974 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c9b397-ac52-4bb4-8964-f7a02dddba61 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:40.740549 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-84c9b397-ac52-4bb4-8964-f7a02dddba61 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:40.828500 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c9b397-ac52-4bb4-8964-f7a02dddba61 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:40.830834 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c9b397-ac52-4bb4-8964-f7a02dddba61 service nova] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:40.849290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 723/1437] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:54:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:54:41.902706 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f969acf9-185d-4ced-9e28-d646fcd35572 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:41.903267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f969acf9-185d-4ced-9e28-d646fcd35572 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:41.903267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f969acf9-185d-4ced-9e28-d646fcd35572 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:41.904613 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 715/1438] 127.0.0.1 () {54 vars in 901 bytes} [Wed Sep 7 03:54:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:41.911112 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fe29d6-8626-4b09-bad3-64d554f233ec None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:41.911602 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fe29d6-8626-4b09-bad3-64d554f233ec None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:41.911701 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fe29d6-8626-4b09-bad3-64d554f233ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:41.986875 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-43fe29d6-8626-4b09-bad3-64d554f233ec None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:41.988078 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-43fe29d6-8626-4b09-bad3-64d554f233ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:41.990109 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:42.085045 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 724/1439] 127.0.0.1 () {58 vars in 998 bytes} [Wed Sep 7 03:54:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:42.234310 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f969acf9-185d-4ced-9e28-d646fcd35572 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:42.235909 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-f969acf9-185d-4ced-9e28-d646fcd35572 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:42.380288 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f969acf9-185d-4ced-9e28-d646fcd35572 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:42.409790 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:42.495450 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbe2d4a-96fe-4d35-b0eb-82c70aa952f3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:42.496055 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbe2d4a-96fe-4d35-b0eb-82c70aa952f3 service nova] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:42.496334 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbe2d4a-96fe-4d35-b0eb-82c70aa952f3 service nova] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:42.496605 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbe2d4a-96fe-4d35-b0eb-82c70aa952f3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:42.498236 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-2cbe2d4a-96fe-4d35-b0eb-82c70aa952f3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:42.586890 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbe2d4a-96fe-4d35-b0eb-82c70aa952f3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:42.588488 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbe2d4a-96fe-4d35-b0eb-82c70aa952f3 service nova] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:42.607271 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 716/1440] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:54:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:54:44.692272 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb65fbe-4427-4394-aae4-3a1a011ca801 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:44.692839 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb65fbe-4427-4394-aae4-3a1a011ca801 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:44.692839 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb65fbe-4427-4394-aae4-3a1a011ca801 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:44.694249 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 725/1441] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 7 03:54:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:44.699589 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec01384d-07c4-4947-904e-d835256cfb69 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:44.700043 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec01384d-07c4-4947-904e-d835256cfb69 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:44.700158 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec01384d-07c4-4947-904e-d835256cfb69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:44.741928 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:44.767097 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-ec01384d-07c4-4947-904e-d835256cfb69 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:44.768927 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-ec01384d-07c4-4947-904e-d835256cfb69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:44.864650 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 717/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 7 03:54:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:45.147622 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb65fbe-4427-4394-aae4-3a1a011ca801 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:45.149161 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-6bb65fbe-4427-4394-aae4-3a1a011ca801 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:45.253035 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb65fbe-4427-4394-aae4-3a1a011ca801 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:45.285568 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:45.326597 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6174a51-358f-4fb6-b7af-830bffd32cfd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:45.327205 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6174a51-358f-4fb6-b7af-830bffd32cfd service neutron] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:45.327480 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6174a51-358f-4fb6-b7af-830bffd32cfd service neutron] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:45.327747 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6174a51-358f-4fb6-b7af-830bffd32cfd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:45.329326 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-b6174a51-358f-4fb6-b7af-830bffd32cfd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:45.436859 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6174a51-358f-4fb6-b7af-830bffd32cfd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:45.438258 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6174a51-358f-4fb6-b7af-830bffd32cfd service neutron] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:45.456893 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 726/1443] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:54:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:54:47.852695 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd964e3-af39-470b-be65-ac6308f6b14f None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:47.853266 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd964e3-af39-470b-be65-ac6308f6b14f None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:47.853266 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd964e3-af39-470b-be65-ac6308f6b14f None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:47.854639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 718/1444] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 7 03:54:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:47.859922 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b140024c-50d9-4c71-be07-e76713cd4e83 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:47.860372 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b140024c-50d9-4c71-be07-e76713cd4e83 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:47.860479 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b140024c-50d9-4c71-be07-e76713cd4e83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:47.925802 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-b140024c-50d9-4c71-be07-e76713cd4e83 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:47.927040 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-b140024c-50d9-4c71-be07-e76713cd4e83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:47.940163 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:48.024276 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 727/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 7 03:54:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:48.312531 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd964e3-af39-470b-be65-ac6308f6b14f None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:48.314218 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-dbd964e3-af39-470b-be65-ac6308f6b14f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:48.418678 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd964e3-af39-470b-be65-ac6308f6b14f None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:48.430849 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:48.492745 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016c5301-dc36-4309-aba9-1079d6111fce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:48.493358 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016c5301-dc36-4309-aba9-1079d6111fce service neutron] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:48.493642 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016c5301-dc36-4309-aba9-1079d6111fce service neutron] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:48.493921 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016c5301-dc36-4309-aba9-1079d6111fce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:48.495601 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-016c5301-dc36-4309-aba9-1079d6111fce service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:48.603988 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016c5301-dc36-4309-aba9-1079d6111fce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:48.605424 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016c5301-dc36-4309-aba9-1079d6111fce service neutron] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:48.623047 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 719/1446] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:54:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:54:51.124296 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad3cdff-a0d1-40d8-82d2-1253656c9ab3 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:51.124930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad3cdff-a0d1-40d8-82d2-1253656c9ab3 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:51.124930 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad3cdff-a0d1-40d8-82d2-1253656c9ab3 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:51.126261 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 728/1447] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 7 03:54:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:51.132209 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5f2f44-0589-40e6-896d-ffd6941ad697 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:51.132626 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5f2f44-0589-40e6-896d-ffd6941ad697 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:51.132756 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5f2f44-0589-40e6-896d-ffd6941ad697 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:51.182405 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:51.199810 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-ce5f2f44-0589-40e6-896d-ffd6941ad697 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:51.201012 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-ce5f2f44-0589-40e6-896d-ffd6941ad697 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:51.297483 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 720/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 7 03:54:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:51.580978 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad3cdff-a0d1-40d8-82d2-1253656c9ab3 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:51.582799 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-fad3cdff-a0d1-40d8-82d2-1253656c9ab3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:51.686413 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad3cdff-a0d1-40d8-82d2-1253656c9ab3 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:51.726290 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:51.760388 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5094f9a0-3007-4427-999b-241b5382f5e7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:51.761004 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5094f9a0-3007-4427-999b-241b5382f5e7 service neutron] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:51.761877 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5094f9a0-3007-4427-999b-241b5382f5e7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:51.762181 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5094f9a0-3007-4427-999b-241b5382f5e7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:51.765828 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-5094f9a0-3007-4427-999b-241b5382f5e7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:51.878059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5094f9a0-3007-4427-999b-241b5382f5e7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:51.880447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5094f9a0-3007-4427-999b-241b5382f5e7 service neutron] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:51.898076 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 729/1449] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:54:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:54:54.173496 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40961faa-9d5f-41c9-ab4e-de638d9089f8 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:54.174042 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40961faa-9d5f-41c9-ab4e-de638d9089f8 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:54.174042 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40961faa-9d5f-41c9-ab4e-de638d9089f8 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:54.175428 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 721/1450] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 7 03:54:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:54.180831 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0acf96a-1b1a-40c3-a53c-e50d0bf562fe None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:54.181266 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0acf96a-1b1a-40c3-a53c-e50d0bf562fe None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:54.181371 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0acf96a-1b1a-40c3-a53c-e50d0bf562fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:54.248571 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-a0acf96a-1b1a-40c3-a53c-e50d0bf562fe None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:54.249800 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-a0acf96a-1b1a-40c3-a53c-e50d0bf562fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:54.270951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:54.346467 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 730/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 7 03:54:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:54.637702 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40961faa-9d5f-41c9-ab4e-de638d9089f8 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:54.639348 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-40961faa-9d5f-41c9-ab4e-de638d9089f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:54.741603 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40961faa-9d5f-41c9-ab4e-de638d9089f8 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:54.761191 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:54.815795 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5383002-25eb-4452-88d5-6fee235e388e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:54.816400 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5383002-25eb-4452-88d5-6fee235e388e service neutron] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:54.816681 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5383002-25eb-4452-88d5-6fee235e388e service neutron] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:54.816953 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5383002-25eb-4452-88d5-6fee235e388e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:54.818580 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-a5383002-25eb-4452-88d5-6fee235e388e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:54.926143 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5383002-25eb-4452-88d5-6fee235e388e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:54.927524 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5383002-25eb-4452-88d5-6fee235e388e service neutron] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:54.945276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 722/1452] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:54:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:54:57.227206 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b1d3a9-632d-44a6-9cf9-1e1177f69d4b None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:57.227777 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b1d3a9-632d-44a6-9cf9-1e1177f69d4b None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:57.227777 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b1d3a9-632d-44a6-9cf9-1e1177f69d4b None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:57.229156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 731/1453] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 7 03:54:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:54:57.234537 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66987e74-be93-41da-b01a-9c85d702a2f5 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:57.234979 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66987e74-be93-41da-b01a-9c85d702a2f5 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:57.235077 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66987e74-be93-41da-b01a-9c85d702a2f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:57.299905 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:57.303187 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-66987e74-be93-41da-b01a-9c85d702a2f5 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:54:57.305396 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-66987e74-be93-41da-b01a-9c85d702a2f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:57.403166 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 723/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 7 03:54:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:54:57.688152 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b1d3a9-632d-44a6-9cf9-1e1177f69d4b None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:54:57.690256 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-d8b1d3a9-632d-44a6-9cf9-1e1177f69d4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:57.791722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b1d3a9-632d-44a6-9cf9-1e1177f69d4b None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:54:57.839756 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:54:57.865375 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05ecb42-f608-4358-bd1d-a1ebf6080293 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:54:57.865985 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05ecb42-f608-4358-bd1d-a1ebf6080293 service neutron] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:54:57.866303 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05ecb42-f608-4358-bd1d-a1ebf6080293 service neutron] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:54:57.866576 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05ecb42-f608-4358-bd1d-a1ebf6080293 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:54:57.868166 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-a05ecb42-f608-4358-bd1d-a1ebf6080293 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:54:57.978754 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05ecb42-f608-4358-bd1d-a1ebf6080293 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:54:57.980114 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05ecb42-f608-4358-bd1d-a1ebf6080293 service neutron] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:54:57.999202 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 732/1455] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:54:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:55:00.210258 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d514cc7-3b02-4d39-a0d9-486426eb83a0 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:00.210805 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d514cc7-3b02-4d39-a0d9-486426eb83a0 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:00.210805 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d514cc7-3b02-4d39-a0d9-486426eb83a0 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:00.212169 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 724/1456] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 7 03:55:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:55:00.217621 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f353f6-523f-4026-8818-29bd597d51a8 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:00.218068 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f353f6-523f-4026-8818-29bd597d51a8 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:00.218208 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f353f6-523f-4026-8818-29bd597d51a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:00.284762 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-95f353f6-523f-4026-8818-29bd597d51a8 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:55:00.285955 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-95f353f6-523f-4026-8818-29bd597d51a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:00.314972 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:00.383869 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 733/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 7 03:55:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:55:00.674386 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d514cc7-3b02-4d39-a0d9-486426eb83a0 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:55:00.675939 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-1d514cc7-3b02-4d39-a0d9-486426eb83a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:00.779652 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d514cc7-3b02-4d39-a0d9-486426eb83a0 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:55:00.808742 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:00.854510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c425c9-24ab-470e-b226-01c8cc40fa52 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:55:00.855117 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c425c9-24ab-470e-b226-01c8cc40fa52 service neutron] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:00.855400 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c425c9-24ab-470e-b226-01c8cc40fa52 service neutron] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:00.855673 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c425c9-24ab-470e-b226-01c8cc40fa52 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:00.857270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-79c425c9-24ab-470e-b226-01c8cc40fa52 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:00.966709 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c425c9-24ab-470e-b226-01c8cc40fa52 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:55:00.968060 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c425c9-24ab-470e-b226-01c8cc40fa52 service neutron] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:55:00.986450 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 725/1458] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:55:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:55:03.243153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e91f85-30d3-461b-9241-1dddd45b0091 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:03.243709 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e91f85-30d3-461b-9241-1dddd45b0091 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:03.243709 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e91f85-30d3-461b-9241-1dddd45b0091 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:03.245090 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 734/1459] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 7 03:55:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:55:03.250341 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e81261-b88a-4e07-a664-c0ac826b10c3 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:03.250777 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e81261-b88a-4e07-a664-c0ac826b10c3 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:03.250930 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e81261-b88a-4e07-a664-c0ac826b10c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:03.315940 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-04e81261-b88a-4e07-a664-c0ac826b10c3 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:55:03.317153 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-04e81261-b88a-4e07-a664-c0ac826b10c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:03.325268 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:03.413886 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 726/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 7 03:55:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:55:03.701722 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e91f85-30d3-461b-9241-1dddd45b0091 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:55:03.703314 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-71e91f85-30d3-461b-9241-1dddd45b0091 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:03.806844 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e91f85-30d3-461b-9241-1dddd45b0091 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:55:03.861855 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:03.880367 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dee4fb2-e63a-44ad-8437-127b7650b6e2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:55:03.880977 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dee4fb2-e63a-44ad-8437-127b7650b6e2 service neutron] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:03.881257 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dee4fb2-e63a-44ad-8437-127b7650b6e2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:03.881533 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dee4fb2-e63a-44ad-8437-127b7650b6e2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:03.883161 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-2dee4fb2-e63a-44ad-8437-127b7650b6e2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:03.992110 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dee4fb2-e63a-44ad-8437-127b7650b6e2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:55:03.993485 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dee4fb2-e63a-44ad-8437-127b7650b6e2 service neutron] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:55:04.011851 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 735/1461] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:55:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:55:06.235365 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183246a2-d0aa-4989-aec8-efb9a53466d1 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:06.235948 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183246a2-d0aa-4989-aec8-efb9a53466d1 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:06.236128 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183246a2-d0aa-4989-aec8-efb9a53466d1 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:06.238090 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 727/1462] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 7 03:55:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:55:06.243616 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de83eb15-3e3f-4e22-af8b-5a80facaf96b None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:06.244053 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de83eb15-3e3f-4e22-af8b-5a80facaf96b None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:06.244163 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de83eb15-3e3f-4e22-af8b-5a80facaf96b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:06.311207 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-de83eb15-3e3f-4e22-af8b-5a80facaf96b None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:55:06.312421 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-de83eb15-3e3f-4e22-af8b-5a80facaf96b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:06.467661 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:06.527540 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 736/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 7 03:55:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 285 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:55:06.832828 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183246a2-d0aa-4989-aec8-efb9a53466d1 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:55:06.834400 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-183246a2-d0aa-4989-aec8-efb9a53466d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:06.936327 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183246a2-d0aa-4989-aec8-efb9a53466d1 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:55:06.972847 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:07.009483 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab3d3ed-c890-4001-ad63-dc37cb413e6f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:55:07.010086 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab3d3ed-c890-4001-ad63-dc37cb413e6f service neutron] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:07.010398 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab3d3ed-c890-4001-ad63-dc37cb413e6f service neutron] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:07.010672 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab3d3ed-c890-4001-ad63-dc37cb413e6f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:07.012260 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-3ab3d3ed-c890-4001-ad63-dc37cb413e6f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:07.119272 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab3d3ed-c890-4001-ad63-dc37cb413e6f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:55:07.120619 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab3d3ed-c890-4001-ad63-dc37cb413e6f service neutron] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:55:07.138008 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 728/1464] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:55:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:55:09.339230 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d27f713-3db4-4c83-9347-792f74c6dc49 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:09.339821 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d27f713-3db4-4c83-9347-792f74c6dc49 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:09.339821 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d27f713-3db4-4c83-9347-792f74c6dc49 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:09.341181 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 737/1465] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 7 03:55:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:55:09.346721 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09827f8-a0b2-4498-b26c-d41c52e7840f None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:09.347129 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09827f8-a0b2-4498-b26c-d41c52e7840f None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:09.347273 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09827f8-a0b2-4498-b26c-d41c52e7840f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:09.413185 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-c09827f8-a0b2-4498-b26c-d41c52e7840f None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:55:09.414439 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-c09827f8-a0b2-4498-b26c-d41c52e7840f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:09.432639 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:09.511739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 729/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 7 03:55:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:55:09.804522 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d27f713-3db4-4c83-9347-792f74c6dc49 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:55:09.806055 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-2d27f713-3db4-4c83-9347-792f74c6dc49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:09.908766 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d27f713-3db4-4c83-9347-792f74c6dc49 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:55:09.973900 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:09.980977 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac90e44a-1782-41c6-be43-1cc5136ce9f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:55:09.981583 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac90e44a-1782-41c6-be43-1cc5136ce9f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:09.981861 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac90e44a-1782-41c6-be43-1cc5136ce9f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:09.982182 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac90e44a-1782-41c6-be43-1cc5136ce9f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:09.983774 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-ac90e44a-1782-41c6-be43-1cc5136ce9f0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:10.092377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac90e44a-1782-41c6-be43-1cc5136ce9f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:55:10.093890 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac90e44a-1782-41c6-be43-1cc5136ce9f0 service neutron] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:55:10.112962 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 738/1467] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:55:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:55:12.293362 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff6f407-7531-4c72-861f-334b9fcf3957 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:12.293843 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff6f407-7531-4c72-861f-334b9fcf3957 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:12.293948 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff6f407-7531-4c72-861f-334b9fcf3957 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:12.295307 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 730/1468] 127.0.0.1 () {54 vars in 917 bytes} [Wed Sep 7 03:55:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:55:12.300941 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6695a18b-17bc-4aa0-9043-c51272656238 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:12.301334 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6695a18b-17bc-4aa0-9043-c51272656238 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:12.301487 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6695a18b-17bc-4aa0-9043-c51272656238 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:12.366757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-6695a18b-17bc-4aa0-9043-c51272656238 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:55:12.367882 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-6695a18b-17bc-4aa0-9043-c51272656238 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:12.417559 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:12.466424 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 739/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Sep 7 03:55:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:55:12.749798 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff6f407-7531-4c72-861f-334b9fcf3957 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:55:12.751494 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-8ff6f407-7531-4c72-861f-334b9fcf3957 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:12.852509 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff6f407-7531-4c72-861f-334b9fcf3957 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:55:12.898287 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:12.925608 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0345ce-b3e5-41aa-881e-e157d507fccc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:55:12.926240 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0345ce-b3e5-41aa-881e-e157d507fccc service neutron] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:12.926521 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0345ce-b3e5-41aa-881e-e157d507fccc service neutron] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:12.926788 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0345ce-b3e5-41aa-881e-e157d507fccc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:12.928370 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-5e0345ce-b3e5-41aa-881e-e157d507fccc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:13.034656 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0345ce-b3e5-41aa-881e-e157d507fccc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:55:13.035993 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0345ce-b3e5-41aa-881e-e157d507fccc service neutron] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:55:13.054020 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 731/1470] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:55:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:55:22.464705 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=22 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:23.051871 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=66 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:33.015372 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c93563-1eca-4e50-9434-46e4be633124 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:33.015984 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c93563-1eca-4e50-9434-46e4be633124 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:33.015984 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c93563-1eca-4e50-9434-46e4be633124 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:33.017308 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 740/1471] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:55:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:55:33.022772 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c94cff1-a785-41bf-b002-7e4f70b5c05d None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:33.023194 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c94cff1-a785-41bf-b002-7e4f70b5c05d None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:33.023333 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c94cff1-a785-41bf-b002-7e4f70b5c05d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:33.089107 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-0c94cff1-a785-41bf-b002-7e4f70b5c05d None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:55:33.090344 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-0c94cff1-a785-41bf-b002-7e4f70b5c05d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:33.187329 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 732/1472] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:55:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:55:33.485983 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec583c61-6cfe-468d-abc5-d124a7f13986 None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:33.486305 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec583c61-6cfe-468d-abc5-d124a7f13986 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:33.486581 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec583c61-6cfe-468d-abc5-d124a7f13986 None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:33.487919 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 741/1473] 10.222.0.27 () {54 vars in 961 bytes} [Wed Sep 7 03:55:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:55:33.492798 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6552407-3c81-4d29-9d54-c95a6bb7ab9d None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:33.493115 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6552407-3c81-4d29-9d54-c95a6bb7ab9d None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:33.493395 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6552407-3c81-4d29-9d54-c95a6bb7ab9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:33.568376 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-b6552407-3c81-4d29-9d54-c95a6bb7ab9d None None] MFA Rules not processed for user `dfa09be69ee3444e94f00cc139bf50e7`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:55:33.569598 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-b6552407-3c81-4d29-9d54-c95a6bb7ab9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:33.589977 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:33.645908 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 733/1474] 10.222.0.27 () {58 vars in 1058 bytes} [Wed Sep 7 03:55:33 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:55:33.653741 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec583c61-6cfe-468d-abc5-d124a7f13986 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:55:33.655300 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-ec583c61-6cfe-468d-abc5-d124a7f13986 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:33.759724 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec583c61-6cfe-468d-abc5-d124a7f13986 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:55:33.831663 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292bde8f-fde7-4d73-b983-80b493989e56 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dfa09be69ee3444e94f00cc139bf50e7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:55:33.832532 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292bde8f-fde7-4d73-b983-80b493989e56 service glance] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:33.832808 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292bde8f-fde7-4d73-b983-80b493989e56 service glance] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:33.833077 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292bde8f-fde7-4d73-b983-80b493989e56 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:33.834680 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-292bde8f-fde7-4d73-b983-80b493989e56 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:33.887951 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:33.942729 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292bde8f-fde7-4d73-b983-80b493989e56 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:55:33.944278 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292bde8f-fde7-4d73-b983-80b493989e56 service glance] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:55:33.962757 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 742/1475] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:55:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:55:36.209872 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9f2299-0b9b-4759-8b74-72764be303d9 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:36.210459 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9f2299-0b9b-4759-8b74-72764be303d9 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:36.210459 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9f2299-0b9b-4759-8b74-72764be303d9 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:36.211789 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 734/1476] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:55:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:55:36.217061 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56a3862-ad38-44c8-8082-430f9eebc4fb None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:36.217448 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56a3862-ad38-44c8-8082-430f9eebc4fb None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:36.217619 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56a3862-ad38-44c8-8082-430f9eebc4fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:36.284714 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-f56a3862-ad38-44c8-8082-430f9eebc4fb None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:55:36.285921 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-f56a3862-ad38-44c8-8082-430f9eebc4fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:36.381433 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 743/1477] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:55:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 07 03:55:36.659660 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9f2299-0b9b-4759-8b74-72764be303d9 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:55:36.661139 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-ce9f2299-0b9b-4759-8b74-72764be303d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:36.785072 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9f2299-0b9b-4759-8b74-72764be303d9 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:55:36.834223 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:36.879520 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dfa9a30-3667-45ca-8a67-3ef1a901c4e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '388f90b795764a26a3cfac5c43c50a76', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:55:36.880127 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfa9a30-3667-45ca-8a67-3ef1a901c4e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:36.880406 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfa9a30-3667-45ca-8a67-3ef1a901c4e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:36.880677 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfa9a30-3667-45ca-8a67-3ef1a901c4e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:36.881314 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dfa9a30-3667-45ca-8a67-3ef1a901c4e9 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:55:36.883104 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: WARNING py.warnings [None req-0dfa9a30-3667-45ca-8a67-3ef1a901c4e9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 07 03:55:36.883104 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: warnings.warn(msg) Sep 07 03:55:36.883104 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: Sep 07 03:55:36.884270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dfa9a30-3667-45ca-8a67-3ef1a901c4e9 admin admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:55:36.902573 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 735/1478] 10.222.0.27 () {56 vars in 1108 bytes} [Wed Sep 7 03:55:36 2022] GET /identity/v3/projects => generated 2553 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 07 03:55:39.095779 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b5dee3-5734-4d9d-8a58-1bfe7d2db9ed None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:39.096434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b5dee3-5734-4d9d-8a58-1bfe7d2db9ed None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:39.096434 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b5dee3-5734-4d9d-8a58-1bfe7d2db9ed None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:39.097658 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 744/1479] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:55:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:55:39.102876 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d690f7-b25a-463f-bcee-5375534cd50c None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:39.103292 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d690f7-b25a-463f-bcee-5375534cd50c None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:39.103436 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d690f7-b25a-463f-bcee-5375534cd50c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:39.170418 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-b9d690f7-b25a-463f-bcee-5375534cd50c None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:55:39.171837 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-b9d690f7-b25a-463f-bcee-5375534cd50c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:39.268813 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 736/1480] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:55:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 07 03:55:39.560629 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b5dee3-5734-4d9d-8a58-1bfe7d2db9ed None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:55:39.562730 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-d1b5dee3-5734-4d9d-8a58-1bfe7d2db9ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:39.572263 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:39.709630 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b5dee3-5734-4d9d-8a58-1bfe7d2db9ed None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:55:39.788959 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:39.822299 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c335f5-74b7-4beb-85a7-c675e31db7db service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '953cb6c24bcc4551b91ce540d18cd751', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:55:39.822842 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c335f5-74b7-4beb-85a7-c675e31db7db service nova] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:39.823121 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c335f5-74b7-4beb-85a7-c675e31db7db service nova] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:39.823402 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c335f5-74b7-4beb-85a7-c675e31db7db service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:39.825002 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-56c335f5-74b7-4beb-85a7-c675e31db7db service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:39.918250 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c335f5-74b7-4beb-85a7-c675e31db7db service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:55:39.919854 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c335f5-74b7-4beb-85a7-c675e31db7db service nova] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:55:39.938415 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 745/1481] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:55:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:55:42.374024 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe12248f-79c3-43f1-92ce-dea8ef975d89 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:42.375114 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe12248f-79c3-43f1-92ce-dea8ef975d89 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:42.375114 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe12248f-79c3-43f1-92ce-dea8ef975d89 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:42.376472 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 737/1482] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:55:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:55:42.381881 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688df5c6-6ef1-4454-b280-b5fc784eb420 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:42.382329 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688df5c6-6ef1-4454-b280-b5fc784eb420 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:42.382501 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688df5c6-6ef1-4454-b280-b5fc784eb420 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:42.450134 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-688df5c6-6ef1-4454-b280-b5fc784eb420 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:55:42.451349 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-688df5c6-6ef1-4454-b280-b5fc784eb420 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:42.496644 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:42.546826 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 746/1483] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:55:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:55:42.845572 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe12248f-79c3-43f1-92ce-dea8ef975d89 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:55:42.847164 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-fe12248f-79c3-43f1-92ce-dea8ef975d89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:42.870532 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:42.950307 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe12248f-79c3-43f1-92ce-dea8ef975d89 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:55:43.023941 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e701c2f0-fa50-428f-bb34-5b8f932a92f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:55:43.024544 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e701c2f0-fa50-428f-bb34-5b8f932a92f5 service neutron] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:43.024824 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e701c2f0-fa50-428f-bb34-5b8f932a92f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:43.025100 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e701c2f0-fa50-428f-bb34-5b8f932a92f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:43.026739 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-e701c2f0-fa50-428f-bb34-5b8f932a92f5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:43.100140 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:43.134532 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e701c2f0-fa50-428f-bb34-5b8f932a92f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:55:43.135954 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e701c2f0-fa50-428f-bb34-5b8f932a92f5 service neutron] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:55:43.154669 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 738/1484] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:55:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:55:45.377234 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79c3b8c-cd9e-4fa0-9fde-ada8adcf9bcc None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:45.377816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79c3b8c-cd9e-4fa0-9fde-ada8adcf9bcc None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:45.377816 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79c3b8c-cd9e-4fa0-9fde-ada8adcf9bcc None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:45.379156 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 747/1485] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:55:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:55:45.384487 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03553c82-9c41-4add-9133-03d767bc867c None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:45.384870 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03553c82-9c41-4add-9133-03d767bc867c None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:45.385044 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03553c82-9c41-4add-9133-03d767bc867c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:45.451032 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-03553c82-9c41-4add-9133-03d767bc867c None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:55:45.452266 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-03553c82-9c41-4add-9133-03d767bc867c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:45.665852 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 739/1486] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:55:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:55:45.957567 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79c3b8c-cd9e-4fa0-9fde-ada8adcf9bcc None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:55:45.959154 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-c79c3b8c-cd9e-4fa0-9fde-ada8adcf9bcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:46.064565 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79c3b8c-cd9e-4fa0-9fde-ada8adcf9bcc None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:55:46.135645 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:46.137512 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c04944b-d709-4177-9903-9c66f1737143 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:55:46.138146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c04944b-d709-4177-9903-9c66f1737143 service neutron] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:46.138699 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c04944b-d709-4177-9903-9c66f1737143 service neutron] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:46.139157 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c04944b-d709-4177-9903-9c66f1737143 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:46.141480 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-7c04944b-d709-4177-9903-9c66f1737143 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:46.249495 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c04944b-d709-4177-9903-9c66f1737143 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:55:46.251237 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c04944b-d709-4177-9903-9c66f1737143 service neutron] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:55:46.269944 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 748/1487] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:55:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:55:48.580138 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cba791-0c73-4e23-a1c4-5fb070a40ba6 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:48.580661 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cba791-0c73-4e23-a1c4-5fb070a40ba6 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:48.580760 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cba791-0c73-4e23-a1c4-5fb070a40ba6 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:48.582033 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 740/1488] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:55:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:55:48.587511 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6217c99-2e8f-4af7-91f9-b901f8520615 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:48.587925 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6217c99-2e8f-4af7-91f9-b901f8520615 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:48.588057 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6217c99-2e8f-4af7-91f9-b901f8520615 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:48.655443 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-c6217c99-2e8f-4af7-91f9-b901f8520615 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:55:48.656690 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-c6217c99-2e8f-4af7-91f9-b901f8520615 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:48.713346 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:48.757584 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 749/1489] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:55:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:55:49.056948 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cba791-0c73-4e23-a1c4-5fb070a40ba6 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:55:49.058581 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-24cba791-0c73-4e23-a1c4-5fb070a40ba6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:49.094489 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:49.160744 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cba791-0c73-4e23-a1c4-5fb070a40ba6 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:55:49.232575 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f09d34f-9891-4e04-89ed-e49e7ccc236b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:55:49.233170 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f09d34f-9891-4e04-89ed-e49e7ccc236b service neutron] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:49.233450 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f09d34f-9891-4e04-89ed-e49e7ccc236b service neutron] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:49.234341 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f09d34f-9891-4e04-89ed-e49e7ccc236b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:49.236884 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-0f09d34f-9891-4e04-89ed-e49e7ccc236b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:49.316763 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:49.347373 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f09d34f-9891-4e04-89ed-e49e7ccc236b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:55:49.348815 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f09d34f-9891-4e04-89ed-e49e7ccc236b service neutron] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:55:49.367182 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 741/1490] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:55:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:55:51.672006 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb4e9bc-1239-479b-a7b1-488d8114c4dc None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:51.672596 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb4e9bc-1239-479b-a7b1-488d8114c4dc None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:51.672596 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb4e9bc-1239-479b-a7b1-488d8114c4dc None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:51.673941 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 750/1491] 10.222.0.27 () {54 vars in 927 bytes} [Wed Sep 7 03:55:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:55:51.679442 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55eb937-a537-4048-96f6-55fd56ca40ff None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:51.679897 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55eb937-a537-4048-96f6-55fd56ca40ff None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:51.680009 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55eb937-a537-4048-96f6-55fd56ca40ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:51.744234 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-a55eb937-a537-4048-96f6-55fd56ca40ff None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:55:51.745428 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-a55eb937-a537-4048-96f6-55fd56ca40ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:51.845758 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 742/1492] 10.222.0.27 () {58 vars in 1024 bytes} [Wed Sep 7 03:55:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:55:52.133136 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb4e9bc-1239-479b-a7b1-488d8114c4dc None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:55:52.134750 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-bfb4e9bc-1239-479b-a7b1-488d8114c4dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:52.236774 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb4e9bc-1239-479b-a7b1-488d8114c4dc None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:55:52.309857 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd090ca-40a1-46cc-9d2e-b4cf3db5b08d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:55:52.310490 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd090ca-40a1-46cc-9d2e-b4cf3db5b08d service neutron] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:55:52.310768 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd090ca-40a1-46cc-9d2e-b4cf3db5b08d service neutron] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:55:52.311038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd090ca-40a1-46cc-9d2e-b4cf3db5b08d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:55:52.313279 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-edd090ca-40a1-46cc-9d2e-b4cf3db5b08d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:55:52.322136 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:55:52.421682 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd090ca-40a1-46cc-9d2e-b4cf3db5b08d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:55:52.423079 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd090ca-40a1-46cc-9d2e-b4cf3db5b08d service neutron] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:55:52.456629 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 751/1493] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:55:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:56:01.843933 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=82 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:56:02.450410 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=50 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:57:06.230076 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32351ac4-e353-47f4-90e9-04f318fcbd2d None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:06.230648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32351ac4-e353-47f4-90e9-04f318fcbd2d None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:06.230648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32351ac4-e353-47f4-90e9-04f318fcbd2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:06.308276 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-32351ac4-e353-47f4-90e9-04f318fcbd2d None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:57:06.309602 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-32351ac4-e353-47f4-90e9-04f318fcbd2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:57:06.404973 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 743/1494] 10.222.0.27 () {58 vars in 958 bytes} [Wed Sep 7 03:57:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:57:06.413092 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddd290f-dbf9-4a75-985d-2a757caf1d17 None None] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:06.413576 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddd290f-dbf9-4a75-985d-2a757caf1d17 None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:06.413693 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddd290f-dbf9-4a75-985d-2a757caf1d17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:06.489354 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.auth.core [None req-eddd290f-dbf9-4a75-985d-2a757caf1d17 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81391) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:57:06.490591 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-eddd290f-dbf9-4a75-985d-2a757caf1d17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:57:06.586468 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 752/1495] 10.222.0.27 () {58 vars in 958 bytes} [Wed Sep 7 03:57:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:57:06.591938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32351ac4-e353-47f4-90e9-04f318fcbd2d None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:57:06.593178 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-32351ac4-e353-47f4-90e9-04f318fcbd2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:57:06.604491 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32351ac4-e353-47f4-90e9-04f318fcbd2d None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:57:06.664941 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:57:06.698640 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4325af32-2644-4129-918f-44281e05fcad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '388f90b795764a26a3cfac5c43c50a76', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:57:06.699250 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4325af32-2644-4129-918f-44281e05fcad admin admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:06.699523 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4325af32-2644-4129-918f-44281e05fcad admin admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:06.700408 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4325af32-2644-4129-918f-44281e05fcad admin admin] PATH_INFO: `/v3/domains` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:06.701385 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4325af32-2644-4129-918f-44281e05fcad admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81392) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 07 03:57:06.701685 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4325af32-2644-4129-918f-44281e05fcad admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:57:06.702728 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: WARNING py.warnings [None req-4325af32-2644-4129-918f-44281e05fcad admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 07 03:57:06.702728 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: warnings.warn(msg) Sep 07 03:57:06.702728 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: Sep 07 03:57:06.703606 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4325af32-2644-4129-918f-44281e05fcad admin admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:57:06.714876 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 744/1496] 10.222.0.27 () {58 vars in 1153 bytes} [Wed Sep 7 03:57:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 07 03:57:06.723664 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddd290f-dbf9-4a75-985d-2a757caf1d17 None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:57:06.733500 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddd290f-dbf9-4a75-985d-2a757caf1d17 None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:57:06.793153 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:57:06.827129 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0afdf2-5c66-41d1-a57f-174157443856 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '388f90b795764a26a3cfac5c43c50a76', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:57:06.827724 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0afdf2-5c66-41d1-a57f-174157443856 admin admin] REQUEST_METHOD: `POST` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:06.828002 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0afdf2-5c66-41d1-a57f-174157443856 admin admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:06.828269 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0afdf2-5c66-41d1-a57f-174157443856 admin admin] PATH_INFO: `/v3/projects` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:06.828897 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0afdf2-5c66-41d1-a57f-174157443856 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:57:06.830660 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: WARNING py.warnings [None req-1e0afdf2-5c66-41d1-a57f-174157443856 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 07 03:57:06.830660 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: warnings.warn(msg) Sep 07 03:57:06.830660 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: Sep 07 03:57:06.831847 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0afdf2-5c66-41d1-a57f-174157443856 admin admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:57:06.888931 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 753/1497] 10.222.0.27 () {60 vars in 1153 bytes} [Wed Sep 7 03:57:06 2022] POST /identity/v3/projects => generated 377 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:57:06.893026 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4325af32-2644-4129-918f-44281e05fcad admin admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:57:06.894550 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-4325af32-2644-4129-918f-44281e05fcad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:57:06.926960 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4325af32-2644-4129-918f-44281e05fcad admin admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:57:07.021441 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc4a9cd-3b84-4213-833d-3a39d2509b57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '388f90b795764a26a3cfac5c43c50a76', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:57:07.022072 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc4a9cd-3b84-4213-833d-3a39d2509b57 admin admin] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:07.022378 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc4a9cd-3b84-4213-833d-3a39d2509b57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:07.022648 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc4a9cd-3b84-4213-833d-3a39d2509b57 admin admin] PATH_INFO: `/v3/users` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:07.023283 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc4a9cd-3b84-4213-833d-3a39d2509b57 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:57:07.024282 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: WARNING py.warnings [None req-bfc4a9cd-3b84-4213-833d-3a39d2509b57 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 07 03:57:07.024282 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: warnings.warn(msg) Sep 07 03:57:07.024282 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: Sep 07 03:57:07.025482 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc4a9cd-3b84-4213-833d-3a39d2509b57 admin admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:57:07.068082 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 745/1498] 10.222.0.27 () {60 vars in 1144 bytes} [Wed Sep 7 03:57:06 2022] POST /identity/v3/users => generated 289 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 07 03:57:07.072124 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0afdf2-5c66-41d1-a57f-174157443856 admin admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:57:07.073607 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-1e0afdf2-5c66-41d1-a57f-174157443856 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:57:07.135150 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0afdf2-5c66-41d1-a57f-174157443856 admin admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:57:07.217184 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:57:07.227920 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b08380c-8290-417f-8075-cd500279edc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '388f90b795764a26a3cfac5c43c50a76', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:57:07.228525 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b08380c-8290-417f-8075-cd500279edc1 admin admin] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:07.228807 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b08380c-8290-417f-8075-cd500279edc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:07.229080 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b08380c-8290-417f-8075-cd500279edc1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:07.229723 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b08380c-8290-417f-8075-cd500279edc1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:57:07.230772 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: WARNING py.warnings [None req-5b08380c-8290-417f-8075-cd500279edc1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 07 03:57:07.230772 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: warnings.warn(msg) Sep 07 03:57:07.230772 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: Sep 07 03:57:07.232261 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b08380c-8290-417f-8075-cd500279edc1 admin admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:57:07.244975 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 754/1499] 10.222.0.27 () {58 vars in 1122 bytes} [Wed Sep 7 03:57:07 2022] GET /identity/v3/roles => generated 1420 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 07 03:57:07.249520 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc4a9cd-3b84-4213-833d-3a39d2509b57 admin admin] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:57:07.251049 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-bfc4a9cd-3b84-4213-833d-3a39d2509b57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:57:07.262387 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc4a9cd-3b84-4213-833d-3a39d2509b57 admin admin] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:57:07.287094 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:57:07.359365 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c90c549-dfe5-443c-8f8d-c657df5efd93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '388f90b795764a26a3cfac5c43c50a76', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:57:07.359967 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c90c549-dfe5-443c-8f8d-c657df5efd93 admin admin] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:07.360241 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c90c549-dfe5-443c-8f8d-c657df5efd93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:07.360509 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c90c549-dfe5-443c-8f8d-c657df5efd93 admin admin] PATH_INFO: `/v3/roles` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:07.361150 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c90c549-dfe5-443c-8f8d-c657df5efd93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:57:07.362172 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: WARNING py.warnings [None req-3c90c549-dfe5-443c-8f8d-c657df5efd93 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 07 03:57:07.362172 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: warnings.warn(msg) Sep 07 03:57:07.362172 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: Sep 07 03:57:07.363056 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c90c549-dfe5-443c-8f8d-c657df5efd93 admin admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:57:07.375200 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 746/1500] 10.222.0.27 () {58 vars in 1122 bytes} [Wed Sep 7 03:57:07 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 07 03:57:07.379521 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b08380c-8290-417f-8075-cd500279edc1 admin admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:57:07.381089 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-5b08380c-8290-417f-8075-cd500279edc1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:57:07.414849 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b08380c-8290-417f-8075-cd500279edc1 admin admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:57:07.508922 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8884995f-cb87-4ba7-bb82-e58c6204958e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '388f90b795764a26a3cfac5c43c50a76', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:57:07.509504 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8884995f-cb87-4ba7-bb82-e58c6204958e admin admin] REQUEST_METHOD: `PUT` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:07.509785 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8884995f-cb87-4ba7-bb82-e58c6204958e admin admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:07.510059 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8884995f-cb87-4ba7-bb82-e58c6204958e admin admin] PATH_INFO: `/v3/projects/8282240734654544a55030071c4660ee/users/154fc943b8d548419e853d63d0c2d442/roles/34235948f8f943a99328516c2171cfff` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:07.532016 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8884995f-cb87-4ba7-bb82-e58c6204958e admin admin] RBAC: Authorizing `identity:create_grant(project_id=8282240734654544a55030071c4660ee, user_id=154fc943b8d548419e853d63d0c2d442, role_id=34235948f8f943a99328516c2171cfff)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:57:07.533042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: WARNING py.warnings [None req-8884995f-cb87-4ba7-bb82-e58c6204958e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 07 03:57:07.533042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: warnings.warn(msg) Sep 07 03:57:07.533042 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: Sep 07 03:57:07.535024 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8884995f-cb87-4ba7-bb82-e58c6204958e admin admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:57:07.558147 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 755/1501] 10.222.0.27 () {60 vars in 1483 bytes} [Wed Sep 7 03:57:07 2022] PUT /identity/v3/projects/8282240734654544a55030071c4660ee/users/154fc943b8d548419e853d63d0c2d442/roles/34235948f8f943a99328516c2171cfff => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:57:07.570675 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c2992f-f566-430d-9966-ac6704bd9c14 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:07.570961 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c2992f-f566-430d-9966-ac6704bd9c14 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:07.571235 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c2992f-f566-430d-9966-ac6704bd9c14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:07.627601 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-c9c2992f-f566-430d-9966-ac6704bd9c14 None None] MFA Rules not processed for user `154fc943b8d548419e853d63d0c2d442`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:57:07.628802 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-c9c2992f-f566-430d-9966-ac6704bd9c14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:57:07.704551 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 747/1502] 10.222.0.27 () {58 vars in 958 bytes} [Wed Sep 7 03:57:07 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:57:07.718142 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86262a0-cd34-40c3-9c96-35260dd858fc None None] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:07.718451 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86262a0-cd34-40c3-9c96-35260dd858fc None None] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:07.718730 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86262a0-cd34-40c3-9c96-35260dd858fc None None] PATH_INFO: `/` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:07.720074 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 756/1503] 10.222.0.27 () {54 vars in 964 bytes} [Wed Sep 7 03:57:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:57:07.725007 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab38907-e467-43c7-9f4f-2a81aec115c5 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:07.725291 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab38907-e467-43c7-9f4f-2a81aec115c5 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:07.725561 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab38907-e467-43c7-9f4f-2a81aec115c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:07.756938 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:57:07.800885 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-5ab38907-e467-43c7-9f4f-2a81aec115c5 None None] MFA Rules not processed for user `989081d67c8945dc8d853bb2769c902a`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:57:07.802045 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-5ab38907-e467-43c7-9f4f-2a81aec115c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:57:07.876819 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 748/1504] 10.222.0.27 () {58 vars in 1061 bytes} [Wed Sep 7 03:57:07 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:57:07.884579 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86262a0-cd34-40c3-9c96-35260dd858fc None None] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:57:07.886187 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-f86262a0-cd34-40c3-9c96-35260dd858fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:57:07.936235 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:57:07.990179 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86262a0-cd34-40c3-9c96-35260dd858fc None None] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:57:08.063196 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e7afad-65d3-447c-ac7c-15d5d4e18907 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '989081d67c8945dc8d853bb2769c902a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:57:08.063737 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e7afad-65d3-447c-ac7c-15d5d4e18907 service cinder] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:08.064021 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e7afad-65d3-447c-ac7c-15d5d4e18907 service cinder] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:08.064297 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e7afad-65d3-447c-ac7c-15d5d4e18907 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:08.065884 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-d6e7afad-65d3-447c-ac7c-15d5d4e18907 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:57:08.163941 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e7afad-65d3-447c-ac7c-15d5d4e18907 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:57:08.165412 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e7afad-65d3-447c-ac7c-15d5d4e18907 service cinder] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:57:08.169550 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:57:08.184340 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 757/1505] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:57:07 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:57:08.229370 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad2e85d-9e90-4bc8-8af1-4ca1a0392fa4 None None] REQUEST_METHOD: `GET` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:08.229660 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad2e85d-9e90-4bc8-8af1-4ca1a0392fa4 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:08.229965 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad2e85d-9e90-4bc8-8af1-4ca1a0392fa4 None None] PATH_INFO: `/` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:08.232267 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 749/1506] 10.222.0.27 () {52 vars in 830 bytes} [Wed Sep 7 03:57:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 07 03:57:08.236064 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e7afad-65d3-447c-ac7c-15d5d4e18907 service cinder] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:57:08.237277 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-d6e7afad-65d3-447c-ac7c-15d5d4e18907 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:57:08.335696 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e7afad-65d3-447c-ac7c-15d5d4e18907 service cinder] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:57:08.429084 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfe69a1-7297-4640-b0a5-1950b3500683 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '388f90b795764a26a3cfac5c43c50a76', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:57:08.429741 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfe69a1-7297-4640-b0a5-1950b3500683 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:08.429989 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfe69a1-7297-4640-b0a5-1950b3500683 admin admin] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:08.430293 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfe69a1-7297-4640-b0a5-1950b3500683 admin admin] PATH_INFO: `/v3/users/154fc943b8d548419e853d63d0c2d442` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:08.442447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfe69a1-7297-4640-b0a5-1950b3500683 admin admin] RBAC: Authorizing `identity:delete_user(user_id=154fc943b8d548419e853d63d0c2d442)` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:57:08.443507 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: WARNING py.warnings [None req-3dfe69a1-7297-4640-b0a5-1950b3500683 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 07 03:57:08.443507 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: warnings.warn(msg) Sep 07 03:57:08.443507 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: Sep 07 03:57:08.444716 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfe69a1-7297-4640-b0a5-1950b3500683 admin admin] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:57:08.454446 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=100 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:57:08.511447 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [None req-3dfe69a1-7297-4640-b0a5-1950b3500683 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '154fc943b8d548419e853d63d0c2d442'} {{(pid=81391) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 07 03:57:08.521628 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [None req-3dfe69a1-7297-4640-b0a5-1950b3500683 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '154fc943b8d548419e853d63d0c2d442'} {{(pid=81391) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 07 03:57:08.522058 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [None req-3dfe69a1-7297-4640-b0a5-1950b3500683 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '154fc943b8d548419e853d63d0c2d442'} {{(pid=81391) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 07 03:57:08.522585 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [None req-3dfe69a1-7297-4640-b0a5-1950b3500683 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '154fc943b8d548419e853d63d0c2d442'} {{(pid=81391) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 07 03:57:08.536936 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.notifications [None req-3dfe69a1-7297-4640-b0a5-1950b3500683 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '154fc943b8d548419e853d63d0c2d442'} {{(pid=81391) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 07 03:57:08.564146 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 758/1507] 10.222.0.27 () {58 vars in 1224 bytes} [Wed Sep 7 03:57:08 2022] DELETE /identity/v3/users/154fc943b8d548419e853d63d0c2d442 => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:57:08.568650 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea11002-7577-4b86-8cd4-5f418b0e3301 None None] REQUEST_METHOD: `POST` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:08.569149 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea11002-7577-4b86-8cd4-5f418b0e3301 None None] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:08.569263 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea11002-7577-4b86-8cd4-5f418b0e3301 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:08.627548 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.auth.core [None req-9ea11002-7577-4b86-8cd4-5f418b0e3301 None None] MFA Rules not processed for user `c81176b6299a493294b869b497bad7f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81392) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 07 03:57:08.628688 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-9ea11002-7577-4b86-8cd4-5f418b0e3301 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:57:08.703108 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=100 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:57:08.716870 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 750/1508] 10.222.0.27 () {58 vars in 958 bytes} [Wed Sep 7 03:57:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 07 03:57:08.727038 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfe69a1-7297-4640-b0a5-1950b3500683 admin admin] Authenticating user token {{(pid=81391) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:57:08.728254 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-3dfe69a1-7297-4640-b0a5-1950b3500683 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:57:08.829470 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfe69a1-7297-4640-b0a5-1950b3500683 admin admin] Validating token access rules against request {{(pid=81391) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:57:08.883917 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:57:08.905170 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6250e147-6a3e-4d32-8d6c-f43b9ac22e56 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '77aac66a6ab44fc1a80ff2fd1572254b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7c4816e95144485e8135e5b8f68c3662', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81391) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:57:08.905781 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6250e147-6a3e-4d32-8d6c-f43b9ac22e56 service neutron] REQUEST_METHOD: `GET` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:08.906060 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6250e147-6a3e-4d32-8d6c-f43b9ac22e56 service neutron] SCRIPT_NAME: `/identity` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:08.906362 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6250e147-6a3e-4d32-8d6c-f43b9ac22e56 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81391) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:08.907964 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.fernet_utils [None req-6250e147-6a3e-4d32-8d6c-f43b9ac22e56 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81391) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:57:09.016377 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6250e147-6a3e-4d32-8d6c-f43b9ac22e56 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:57:09.017813 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6250e147-6a3e-4d32-8d6c-f43b9ac22e56 service neutron] RBAC: Authorization granted {{(pid=81391) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:57:09.036181 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: [pid: 81391|app: 0|req: 759/1509] 10.222.0.27 () {60 vars in 1365 bytes} [Wed Sep 7 03:57:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 07 03:57:09.382993 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea11002-7577-4b86-8cd4-5f418b0e3301 None None] Authenticating user token {{(pid=81392) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 07 03:57:09.384313 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.fernet_utils [None req-9ea11002-7577-4b86-8cd4-5f418b0e3301 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81392) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 07 03:57:09.396510 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea11002-7577-4b86-8cd4-5f418b0e3301 None None] Validating token access rules against request {{(pid=81392) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 07 03:57:09.494245 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044a1324-9c8a-40ce-ae4e-9421808e1759 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c81176b6299a493294b869b497bad7f4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '388f90b795764a26a3cfac5c43c50a76', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81392) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 07 03:57:09.494870 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044a1324-9c8a-40ce-ae4e-9421808e1759 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 07 03:57:09.495145 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044a1324-9c8a-40ce-ae4e-9421808e1759 admin admin] SCRIPT_NAME: `/identity` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 07 03:57:09.495411 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044a1324-9c8a-40ce-ae4e-9421808e1759 admin admin] PATH_INFO: `/v3/projects/8282240734654544a55030071c4660ee` {{(pid=81392) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 07 03:57:09.505091 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044a1324-9c8a-40ce-ae4e-9421808e1759 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8282240734654544a55030071c4660ee)` {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 07 03:57:09.506103 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: WARNING py.warnings [None req-044a1324-9c8a-40ce-ae4e-9421808e1759 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 07 03:57:09.506103 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: warnings.warn(msg) Sep 07 03:57:09.506103 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: Sep 07 03:57:09.507312 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044a1324-9c8a-40ce-ae4e-9421808e1759 admin admin] RBAC: Authorization granted {{(pid=81392) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 07 03:57:09.564471 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [None req-044a1324-9c8a-40ce-ae4e-9421808e1759 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8282240734654544a55030071c4660ee', 'request_id': 'req-044a1324-9c8a-40ce-ae4e-9421808e1759'} {{(pid=81392) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 07 03:57:09.574270 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [None req-044a1324-9c8a-40ce-ae4e-9421808e1759 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8282240734654544a55030071c4660ee', 'request_id': 'req-044a1324-9c8a-40ce-ae4e-9421808e1759'} {{(pid=81392) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 07 03:57:09.580858 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [None req-044a1324-9c8a-40ce-ae4e-9421808e1759 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8282240734654544a55030071c4660ee', 'request_id': 'req-044a1324-9c8a-40ce-ae4e-9421808e1759'} {{(pid=81392) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 07 03:57:09.610798 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [None req-044a1324-9c8a-40ce-ae4e-9421808e1759 admin admin] The token cache is being invalidate because project 8282240734654544a55030071c4660ee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81392) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 07 03:57:09.611085 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [None req-044a1324-9c8a-40ce-ae4e-9421808e1759 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81392) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 07 03:57:09.611541 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG keystone.notifications [None req-044a1324-9c8a-40ce-ae4e-9421808e1759 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81392) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 07 03:57:09.612935 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: [pid: 81392|app: 0|req: 751/1510] 10.222.0.27 () {58 vars in 1233 bytes} [Wed Sep 7 03:57:09 2022] DELETE /identity/v3/projects/8282240734654544a55030071c4660ee => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 07 03:57:19.033825 dv-cinder-856002-2-iscsi devstack@keystone.service[81391]: DEBUG dbcounter [-] [81391] Writing DB stats keystone:SELECT=62 {{(pid=81391) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 07 03:57:19.608775 dv-cinder-856002-2-iscsi devstack@keystone.service[81392]: DEBUG dbcounter [-] [81392] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81392) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}